Iceberg's Metadata Layer

Iceberg's Metadata Layer in Detail

The metadata JSON file is the root of everything. After the load of Writing Order Data and the delete and rollback of Data and Delete Files, the orders table's current metadata file summarizes as follows:

meta_json.sh: the orders table's current metadata file, summarizedShell

# The orders table's current metadata file, found through the catalog's pointer, via jq.
T=localhost:31181/v1/namespaces/booknest/tables/orders
LOC=$(curl -s $T | jq -r '."metadata-location"')
echo "${LOC##*/}"
mc cat "lake/${LOC#s3://}" | jq -r '
  "format-version \(."format-version"), last-sequence-number \(."last-sequence-number")",
  (."partition-specs"[] | "spec \(."spec-id"): \(.fields[0] | "\(.name) = \(.transform)" +
    "(field \(."source-id"))")"),
  (.snapshots[] | "seq \(."sequence-number") snapshot \(."snapshot-id") parent " +
    "\(."parent-snapshot-id")"),
  "main -> \(.refs.main."snapshot-id"), metadata-log: \(."metadata-log" | length) files"'

Output

00004-361f48b0-4e46-4e1f-b804-e07bab6ed962.metadata.json
format-version 2, last-sequence-number 3
spec 0: order_ts_month = month(field 3)
seq 1 snapshot 3742401330379284395 parent null
seq 2 snapshot 916038210498707551 parent 3742401330379284395
seq 3 snapshot 996694203951870685 parent 916038210498707551
main -> 916038210498707551, metadata-log: 4 files

Schemas and partition specs are lists with IDs, so files written under older ones stay readable after evolution; field 3 is order_ts. Refs name snapshots: main is the branch queries read, and it points at sequence 2, not 3, because the delete was rolled back (Branches and Tags adds branches and tags). The metadata log lists earlier metadata files (100 kept by default), and the sequence number orders data and delete files.

MENU