Iceberg with Spark and Trino

Installing Iceberg with Spark and Trino

Iceberg 129 is a library, but it needs a catalog: here l1-iceberg-rest on port 31181, Iceberg's test image apache/iceberg-rest-fixture (JDBC over SQLite 4,756 , started by demos/ch08/iceberg/run_rest.sh; Iceberg Catalogs covers production catalogs). Spark 129 needs two jars from Maven 129 Central, the runtime and the AWS 24 bundle for S3FileIO:

lake.py: a Spark session for the lakehouse, imported by every Spark listingPython

"""A local Spark 4.1 session wired to BookNest's lakehouse: Iceberg REST catalog + MinIO."""
from pyspark.sql import SparkSession
JARS = "/home/dev/v7-l1/ch08/jars/"                 # fetched once from Maven Central
C = "spark.sql.catalog.lake"                        # the catalog's name in SQL: lake
spark = (SparkSession.builder.master("local[2]").appName("booknest-lakehouse")
    .config("spark.jars", JARS + "iceberg-spark-runtime-4.1_2.13-1.12.0.jar,"
                          + JARS + "iceberg-aws-bundle-1.12.0.jar")
    .config("spark.sql.extensions",
            "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions")
    .config(C, "org.apache.iceberg.spark.SparkCatalog")
    .config(f"{C}.type", "rest")
    .config(f"{C}.uri", "http://localhost:31181")
    .config(f"{C}.io-impl", "org.apache.iceberg.aws.s3.S3FileIO")
    .config(f"{C}.s3.endpoint", "http://localhost:31900")
    .config(f"{C}.s3.path-style-access", "true")
    .config(f"{C}.s3.access-key-id", "booknest-admin")
    .config(f"{C}.s3.secret-access-key", "booknest-secret-2026")
    .config(f"{C}.client.region", "us-east-1")
    .config("spark.sql.defaultCatalog", "lake")
    .config("spark.sql.session.timeZone", "UTC")    # this host runs at UTC+8
    .config("spark.ui.port", "31040")
    .getOrCreate())
spark.sparkContext.setLogLevel("ERROR")

The extensions add Iceberg's SQL (CALL procedures, partition evolution, branches). spark.jars.packages could download the jars, but Ivy failed TLS checks on this host, so they were fetched once and checked against Maven's SHA-1 files. Trino 483 403,499 needs no jars: its built-in connector reads a catalog file naming the same REST URI and MinIO 30,943 (Trino with Iceberg), and its join of orders and order_items (demos/ch08/trino/crosscheck.sql) returns 100,000 orders, 137,944 lines and a non-cancelled gross of 3,303,427.30.

MENU