Optimistic Concurrency

Optimistic Concurrency and Atomic Commits

Table formats take no locks while a job runs. A writer reads the current metadata, writes data and metadata files optimistically, and then asks the catalog to swap the pointer only if it still names the version it started from; otherwise, says the Iceberg 129 specification, "the writer must retry the update based on the new current version". The REST catalog sends that condition as requirements. Two writers race to add a column:

occ.sh: two schema changes race; the catalog accepts oneShell

# Two writers race to change one table through the Iceberg REST catalog; only one base can win.
T=http://localhost:31181/v1/namespaces/booknest/tables
J=(-H 'Content-Type: application/json')
curl -s -X POST $T "${J[@]}" -o /dev/null -d '{"name": "occ_demo", "schema": {"type": "struct",
  "fields": [{"id": 1, "name": "id", "type": "int", "required": true}]}}'
meta() { curl -s $T/occ_demo | jq -r '"schema \(.metadata."current-schema-id"), " +
  (."metadata-location" | split("/") | last)'; }
echo "both writers read: $(meta)"
commit() {   # $1 = writer, $2 = column to add, assuming schema 0 (what both writers read)
  curl -s -X POST $T/occ_demo "${J[@]}" -o occ.json -w "$1 adds $2: HTTP %{http_code}\n" -d '{
    "requirements": [{"type": "assert-current-schema-id", "current-schema-id": 0}],
    "updates": [{"action": "add-schema", "schema": {"type": "struct", "schema-id": 1,
        "fields": [{"id": 1, "name": "id", "type": "int", "required": true},
                   {"id": 2, "name": "'$2'", "type": "string", "required": false}]}},
      {"action": "set-current-schema", "schema-id": -1}]}'
  jq -r '.error.message // empty' occ.json; }
commit writer-A format
commit writer-B edition
echo "table now: $(meta)"
curl -s -X DELETE "$T/occ_demo?purgeRequested=true" -o /dev/null

Output

both writers read: schema 0, 00000-aa4eb8fc-4062-474a-889c-cbd8a3b29504.metadata.json
writer-A adds format: HTTP 200
writer-B adds edition: HTTP 409
Requirement failed: current schema changed: expected id 0 != 1
table now: schema 1, 00001-bb2a1a21-f304-4fe2-87dd-e5974aca2f9f.metadata.json

Writer B got 409 Conflict instead of silently overwriting A's change; data commits assert the branch's snapshot ID the same way. On a conflict, Iceberg's writers refresh and retry (4 times by default, commit.retry.num-retries) after checking that the winner did not touch the same data: appends succeed on retry, while two overwrites of one partition fail with a ValidationException your job must handle.

MENU