UniForm

UniForm

UniForm and Reading Delta Tables as Iceberg

UniForm (Universal Format) makes a Delta table also an Iceberg 129 table: after each commit, the delta-iceberg module writes Iceberg metadata beside the Delta log, pointing at the same Parquet 129 files. It needs column mapping, IcebergCompatV2, no deletion vectors and a Hive 129 metastore (here Spark 129 's embedded one); Iceberg clients only read. Delta 4.4's delta-iceberg supports Spark 4.1 but not 4.2.

uniform.py: a UniForm table written by Spark, read by DuckDB as IcebergPython

"""UniForm: Spark writes a Delta table that also carries Iceberg metadata; DuckDB reads it."""
import glob, time
import duckdb
from dspark import spark
spark.sql("CREATE DATABASE IF NOT EXISTS booknest")
spark.sql("""CREATE TABLE booknest.orders_uniform USING delta
    TBLPROPERTIES ('delta.enableIcebergCompatV2' = 'true',
                   'delta.universalFormat.enabledFormats' = 'iceberg')
    AS SELECT * FROM delta.`/home/dev/v7-l1/ch08/delta/orders` VERSION AS OF 0 LIMIT 0""")
spark.sql("""INSERT INTO booknest.orders_uniform           -- converted after this commit
             SELECT * FROM delta.`/home/dev/v7-l1/ch08/delta/orders` VERSION AS OF 0""")
P = spark.sql("DESCRIBE DETAIL booknest.orders_uniform").first().location.replace("file:", "")
while not (meta := sorted(glob.glob(f"{P}/metadata/*.metadata.json"))):
    time.sleep(1)                                  # Iceberg metadata is written asynchronously
print("DuckDB, as Iceberg:", duckdb.sql(f"""SELECT count(*), sum(gross) FILTER (WHERE
    status <> 'cancelled') FROM iceberg_scan('{meta[-1]}')""").fetchone())

Output

DuckDB, as Iceberg: (100000, Decimal('3303427.30'))

DuckDB 61,228 's Iceberg reader, which knows nothing about Delta, returned the baseline from the same files. A plain CREATE TABLE ... AS SELECT produced no Iceberg metadata here, so the script inserts after creating the table, and the wait loop covers the asynchronous conversion.

MENU