Snapshots

Snapshots

Snapshots as Immutable Table Versions

A snapshot is a table's complete state at one commit: an ID, a parent, an operation (append, overwrite, delete, replace), a summary and a manifest list. Nothing it references is modified, so old snapshots stay readable until expired (Compaction and Expiry). This script built the books table, with Iceberg with Spark and Trino's Spark 129 session:

books.py: one insert, one price change, two snapshots, two readable versionsPython

"""BookNest's six books as an Iceberg table: one insert, one price change, two snapshots."""
from pyspark.sql import functions as F
from lake import spark
spark.sql("CREATE NAMESPACE IF NOT EXISTS booknest")
spark.sql("""CREATE TABLE booknest.books (book_id INT, title STRING, author STRING,
             genre STRING, price DECIMAL(9,2), year INT) USING iceberg""")
raw = spark.read.option("multiLine", True).json(
    "/mnt/d/Books/Data Engineering/demos/ch03/data/books.json")
books = raw.select(F.explode("books").alias("b")).select(
    F.col("b.id").cast("int").alias("book_id"), "b.title", "b.author", "b.genre",
    F.col("b.price").cast("decimal(9,2)").alias("price"),
    F.col("b.year").cast("int").alias("year"))
books.writeTo("booknest.books").append()                                  # snapshot 1
spark.sql("UPDATE booknest.books SET price = 12.99 WHERE book_id = 1")   # snapshot 2
snaps = spark.sql("""SELECT snapshot_id, parent_id, operation,
    summary['added-data-files'] AS added, summary['deleted-data-files'] AS removed
    FROM booknest.books.snapshots ORDER BY committed_at""")
snaps.show()
first = snaps.first().snapshot_id                                       # time travel:
spark.sql(f"""SELECT (SELECT price FROM booknest.books VERSION AS OF {first}
                      WHERE book_id = 1) AS first_price, price AS current_price
              FROM booknest.books WHERE book_id = 1""").show()

Output

+-------------------+-------------------+---------+-----+-------+
|        snapshot_id|          parent_id|operation|added|removed|
+-------------------+-------------------+---------+-----+-------+
|9156398221527800842|               NULL|   append|    1|   NULL|
|1796896758330598411|9156398221527800842|overwrite|    1|      1|
+-------------------+-------------------+---------+-----+-------+
+-----------+-------------+
|first_price|current_price|
+-----------+-------------+
|      14.99|        12.99|
+-----------+-------------+

The UPDATE became an overwrite that added one file and removed one, and VERSION AS OF read the old price. Snapshot IDs are random; order comes from the parent chain and each snapshot's sequence number (Data and Delete Files).

MENU