The Iceberg REST Catalog

The Iceberg REST Catalog Specification

Early Iceberg 129 shipped one Java client per catalog (Hive 129 , Glue, JDBC, Nessie), which every engine had to bundle and every Python or Rust reader had to reimplement. The REST catalog specification, an OpenAPI document at open-api/rest-catalog-open-api.yaml in apache/iceberg (https://github.com/apache/iceberg 9,293 ), replaced that with one HTTP protocol: engines ship one client, and catalogs compete on the server side. Spark 129 , Trino 403,499 , Flink 129 , DuckDB 61,228 , PyIceberg and Snowflake all speak it.

The Iceberg REST protocol: the catalog hands out metadata and access, the engine moves the data

The catalog never touches row data: the engine loads a table's metadata and a config map, reads and writes storage itself, and commits requirements and updates (Optimistic Concurrency). Iceberg 1.12.0's specification has 23 paths, including views, multi-table transactions/commit, server-side scan planning, and two ways to reach storage. A client that sends X-Iceberg-Access-Delegation: vended-credentials receives short-lived storage credentials; with remote-signing it asks the catalog's sign endpoint to sign each S3 request. Either way, engines hold no long-lived keys and the catalog can enforce permissions. The fixture answers calls 1 and 2 like this:

rest_calls.sh: the REST protocol by handShell

# The REST protocol by hand: the server's config, then what loadTable returns.
R=localhost:31181/v1
curl -s $R/config | jq -c '{defaults, overrides, endpoints: (.endpoints | length)}'
curl -s $R/namespaces/booknest/tables/books | jq -c '{"metadata-location":
  (."metadata-location" | split("/") | last), config, metadata: (.metadata | keys | length)}'

Output

{"defaults":{},"overrides":{"namespace-separator":"%2E"},"endpoints":31}
{"metadata-location":"00002-faaa3e6f-b9bd-47c4-bcd3-6eb904d135a5.metadata.json","config":null,"
  metadata":21}

The fixture returns no config, so engines use the MinIO 30,943 keys in lake.py. The specification still moves: 1.12.0 added function endpoints and deprecated the signer.endpoint property in favor of the standard sign path, and oauth/tokens has been deprecated since 1.6.0.

MENU