Metadata and Manifests

Metadata Files, Manifest Lists and Manifests

Iceberg 129 keeps that list as a tree of immutable files. After books.py (Snapshots) inserted the six books and changed one price, the books table's directory holds:

tree.sh: every object of the books table after two commitsShell
# Every object of the books table after two commits (UUIDs and snapshot IDs shortened).
U='[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}'
mc ls -r lake/warehouse/booknest/books/ | sed -E "s/^\[[^]]*\] *//; s/$U/<uuid>/g;
  s/snap-([0-9]{4})[0-9]+/snap-\1.../"
Output
2.0KiB STANDARD data/00000-1-<uuid>-0-00001.parquet
2.0KiB STANDARD data/00000-5-<uuid>-0-00001.parquet
970B STANDARD metadata/00000-<uuid>.metadata.json
2.0KiB STANDARD metadata/00001-<uuid>.metadata.json
3.1KiB STANDARD metadata/00002-<uuid>.metadata.json
7.2KiB STANDARD metadata/<uuid>-m0.avro
7.2KiB STANDARD metadata/<uuid>-m1.avro
7.2KiB STANDARD metadata/<uuid>-m0.avro
4.4KiB STANDARD metadata/snap-1796...-1-<uuid>.avro
4.3KiB STANDARD metadata/snap-9156...-1-<uuid>.avro

A metadata file (00002-....metadata.json) is the table's root at one moment: schemas, partition specs, properties and the snapshots, each pointing to its manifest list (snap-....avro). The manifest list names the snapshot's manifests with partition ranges so planners can skip whole manifests, and each manifest (...-m0.avro) lists data files with partition values, row counts, column statistics and a status (added, existing or deleted). Every commit writes a new metadata file; 00000 is the empty table of CREATE TABLE.

The books table's metadata tree after an insert and an update
The books table's metadata tree after an insert and an update

The update never touched file A: it wrote file B, and snapshot 1796's manifests record A as deleted and B as added, while snapshot 9156 still points at A.