S3 Tables and Lake Formation

AWS S3 Tables, Glue and Lake Formation

AWS 24 splits the lakehouse into three services. Amazon S3 24 Tables adds a new bucket type, the table bucket, whose tables are Iceberg 129 (version 3 included) and which S3 maintains itself: "compaction, snapshot management, and unreferenced file removal", configurable per table. The AWS Glue 24 Data Catalog is the catalog that Athena, Redshift 24 , EMR and Glue jobs share; integrated with a table bucket, it lists the tables and serves them to any engine through the Glue Iceberg REST endpoint. AWS Lake Formation adds fine-grained permissions (tables, columns, rows) on top of Glue and enforces them in the integrated engines.

A table bucket also has its own Iceberg REST endpoint, so BookNest's existing PyIceberg or Spark 129 code moves by changing catalog properties only. Requests are signed with AWS Signature Version 4 under the signing name s3tables, and the warehouse becomes the bucket's ARN (from the S3 Tables documentation, not run here):

Pointing PyIceberg at an S3 table bucket (from documentation, not run here)

from pyiceberg.catalog import load_catalog
lake = load_catalog("booknest", **{
    "type": "rest",
    "uri": "https://s3tables.us-east-1.amazonaws.com/iceberg",
    "warehouse": "arn:aws:s3tables:us-east-1:111122223333:bucket/booknest",
    "rest.sigv4-enabled": "true",
    "rest.signing-name": "s3tables",
    "rest.signing-region": "us-east-1"})

The endpoint has limits worth knowing: single-level namespaces only, no CREATE TABLE AS SELECT (no staged creates), drops always purge, and no OAuth. Pricing (us-east-1) is all metered: $0.0265 per GB-month of table storage (versus $0.023 in a general-purpose bucket), the usual request prices, $0.025 per 1,000 objects monitored, and compaction at $0.002 per 1,000 objects plus $0.005 per GB processed. Glue's Data Catalog stores the first million objects free, and Lake Formation permissions cost nothing; compute is Athena, EMR or Glue at $0.44 per DPU-hour (Costs Compared).

MENU