Delta Concurrency Control

Delta Lake's Concurrency Control

Delta's optimistic commit succeeds only if its writer is first to create the next log file, say ...002.json. Object storage must therefore offer put-if-absent: MinIO 30,943 and Cloudflare 2 R2 do (Consistency and Versioning), which delta-rs uses when given aws_conditional_put: etag; on AWS 24 S3 its documentation still recommends a DynamoDB 24 lock table.

A writer that loses the race checks the commits it missed for logical conflicts (removed files it read, or added files its predicate would match) and retries at the next version only if there are none. The default isolation level, WriteSerializable, lets blind appends through. In demos/ch08/delta/conflict.py, two delta-rs writers update different books in a one-file table from version 0: A commits version 1, B fails ("a concurrent transactions added new data") because A rewrote the file B read, and B reloads and commits version 2.