Consistency and Versioning

Consistency, Versioning and Multipart Uploads

Consistency. Until December 2020, S3 was only eventually consistent for overwrites and listings, so a job could list a prefix right after a write and miss the new file. Since 1 December 2020 it is strongly consistent for all operations, including listings; MinIO 30,943 commits each write to a quorum of drives and behaves the same way.

Conditional writes. S3 added If-None-Match on 20 August 2024 (write only if the key does not exist) and If-Match on 25 November 2024 (write only if the ETag is unchanged): put-if-absent and compare-and-swap on the store itself, which a log-based format such as Delta Lake 237,929 can commit with, no lock service needed.

Versioning and multipart uploads. A versioned bucket keeps overwritten and deleted objects as older versions. Large objects go up in parts (up to 10,000 of 5 MiB to 5 GiB on S3) that upload in parallel and become visible only when the upload completes:

versioning.sh: versions, a conditional write race and a multipart uploadShell

# Versioning, conditional writes and a multipart upload against l1-minio, with mc and curl.
mc version enable lake/booknest-demo
echo '{"book_id": 1, "price": 14.99}' | mc pipe -q lake/booknest-demo/price.json >/dev/null
echo '{"book_id": 1, "price": 12.99}' | mc pipe -q lake/booknest-demo/price.json >/dev/null
OLD=$(mc ls --versions --json lake/booknest-demo/price.json |
  jq -r 'select(.versionOrdinal == 1).versionId')
mc cat --vid "$OLD" lake/booknest-demo/price.json
# If-None-Match: * writes only if the key does not exist yet -> the second writer gets 412.
for who in writer-A writer-B; do
  curl -s --user booknest-admin:booknest-secret-2026 --aws-sigv4 "aws:amz:us-east-1:s3" \
    -X PUT -H "If-None-Match: *" --data-binary "$who" -o /dev/null -w "$who -> %{http_code}\n" \
    http://localhost:31900/booknest-demo/commit-00001.json
done
# A 23.6 MB file goes up as a multipart upload; the ETag's suffix counts the parts.
mc cp -q "/mnt/d/Books/Data Engineering/demos/ch03/data/orders.jsonl" lake/booknest-demo/ \
  >/dev/null
mc stat --json lake/booknest-demo/orders.jsonl | jq -r '"\(.size) bytes, ETag \(.etag)"'

Output

lake/booknest-demo versioning is enabled
{"book_id": 1, "price": 14.99}
writer-A -> 200
writer-B -> 412
23575414 bytes, ETag 1f52af42951055bef1e65d8607f3a37b-2

The overwritten price is still readable by version ID; the second writer lost the race with 412 Precondition Failed instead of silently replacing the first file; and the multipart ETag is an MD5 of the parts' MD5s, -2 for a 16 MiB part plus the remainder, so never compare it with a local md5sum. Iceberg 129 never overwrites an object, so keep bucket versioning for raw landing buckets and leave it off for table storage.

MENU