The Delta Log

The Delta Log

A Delta table is a folder of Parquet 129 files plus _delta_log/, an ordered list of commits named 00000000000000000000.json, ...001.json and so on. Each commit holds one JSON action per line: protocol (the reader and writer features required), metaData (schema, partition columns, properties), add and remove (data files, with statistics), and commitInfo. Replaying the actions gives the table's state; every 10 commits a Parquet checkpoint saves the replay, named in _last_checkpoint. Here is the log of the orders table written in Delta Tables with Spark:

delta_log.sh: what the orders table's two commits containShell

# The orders table's Delta log: one JSON file per commit, one action per line.
L=/home/dev/v7-l1/ch08/delta/orders/_delta_log
for v in 0 1; do                                   # which actions each commit holds
  f=$L/$(printf %020d $v).json
  echo "commit $v: $(jq -r 'keys[0]' $f | sort | uniq -c | xargs)"
done
jq -c 'select(.add) | .add | {path: .path[0:40], size, deletionVector,
  stats: (.stats | fromjson | {numRecords, minTs: .minValues.order_ts[0:10]})}' \
  $L/00000000000000000000.json | head -1
jq -c 'select(.protocol) | .protocol' $L/00000000000000000000.json
jq -c 'select(.add) | .add | {path: .path[0:40], deletionVector: .deletionVector |
  {storageType, cardinality}}' $L/00000000000000000001.json

Output

commit 0: 18 add 1 commitInfo 1 metaData 1 protocol
commit 1: 1 add 1 commitInfo 1 remove
{"path":"order_month=2025-01/part-00000-928e7348-","size":117849,"deletionVector":null,"stats":
  {"numRecords":5713,"minTs":"2025-01-01"}}
{"minReaderVersion":3,"minWriterVersion":7,"readerFeatures":["deletionVectors"],"writerFeatures
  ":["deletionVectors","invariants","appendOnly"]}
{"path":"order_month=2026-06/part-00000-b45a417c-","deletionVector":{"storageType":"u","cardina
  lity":4}}

Commit 0 added 18 files with statistics. Commit 1, a delete of four orders, removed June's file and re-added the same file with a deletion vector marking four rows: Delta's equivalent of Iceberg 129 's position deletes (Data and Delete Files). Reader version 3 and writer version 7 mean "check the feature lists": an engine that lacks deletionVectors must refuse the table.

MENU