Inspecting Metadata Files

Inspecting Metadata and Manifest Files Directly

Manifest lists and manifests are Avro 129 files, so fastavro (Protocol Buffers) can follow the chain by hand. The REST catalog returns the current metadata JSON with the table; the script then reads the manifest list and the first entry of a manifest, decoding statistics as the spec stores them (integers and timestamps little-endian):

inspect_meta.py: from the catalog down to one data file's statisticsPython

"""Follow the orders metadata by hand: catalog -> manifest list -> manifest -> statistics."""
import io, json, subprocess, urllib.request
from datetime import datetime, timezone
import fastavro
def avro(uri):                                  # read an s3:// Avro object through mc's alias
    raw = subprocess.run(["mc", "cat", uri.replace("s3://", "lake/")], capture_output=True)
    return list(fastavro.reader(io.BytesIO(raw.stdout)))
url = "http://localhost:31181/v1/namespaces/booknest/tables/orders"
meta = json.load(urllib.request.urlopen(url))["metadata"]      # the current metadata JSON
cur = next(s for s in meta["snapshots"] if s["snapshot-id"] == meta["current-snapshot-id"])
le = lambda b: int.from_bytes(b, "little")      # ints and timestamps: little-endian
for m in (manifests := avro(cur["manifest-list"])):
    p = m["partitions"][0]                      # month = months since 1970-01
    print(f'{m["manifest_path"][-12:]} content={m["content"]} seq={m["sequence_number"]} '
          f'added={m["added_files_count"]} '
          f'months {le(p["lower_bound"])}..{le(p["upper_bound"])}')
f = avro(manifests[0]["manifest_path"])[0]["data_file"]          # first file of a manifest
ts = lambda b: datetime.fromtimestamp(le(b) / 1e6, timezone.utc).strftime("%Y-%m-%d %H:%M:%S")
lo, hi = ({kv["key"]: kv["value"] for kv in f[k]} for k in ("lower_bounds", "upper_bounds"))
print(f["partition"], f["record_count"], "rows,", f["file_size_in_bytes"], "bytes")
print("order_ts (field 3) from", ts(lo[3]), "to", ts(hi[3]))

Output

8c69-m0.avro content=0 seq=2 added=6 months 672..677
3277-m0.avro content=0 seq=1 added=12 months 660..671
{'order_ts_month': 672} 5643 rows, 56718 bytes
order_ts (field 3) from 2026-01-01 00:01:58 to 2026-01-31 23:58:30

The current snapshot reuses the 2025 manifest (sequence 1) unchanged beside its own 2026 one, and their month ranges let a planner skip a whole manifest. Each entry's column bounds let a filter such as order_ts >= '2026-02-01' skip January's file unopened: Apache Parquet In Depth's predicate pushdown, moved up to whole files.

MENU