Data and Delete Files

Data Files, Delete Files and Positional Deletes

Data files hold rows, in Parquet 129 by default. To change immutable files, a DELETE, UPDATE or MERGE either rewrites every affected file (copy-on-write, the default, as the books table's UPDATE did) or writes small delete files that readers apply on the fly (merge-on-read, set by write.delete.mode and its siblings; compaction folds them in later, Compaction and Expiry). A position delete names a data file and row positions, an equality delete names column values (for streaming upserts), and version 3 stores position deletes as deletion vectors in Puffin files (42 bytes for one row in a test here). Deleting the four pending orders from the merge-on-read orders table writes a position delete file:

deletes.py: a merge-on-read delete, its delete file, and a rollbackPython

"""Merge-on-read delete of the 4 pending orders, a look at the delete file, then a rollback."""
from lake import spark
q = lambda sql: spark.sql(sql).show(truncate=False)
before = spark.table("booknest.orders.refs").where("name = 'main'").first().snapshot_id
spark.sql("DELETE FROM booknest.orders WHERE status = 'pending'")
q("""SELECT content, file_format, partition.order_ts_month AS month, record_count,
            regexp_extract(file_path, '[^/]+$', 0) AS file
     FROM booknest.orders.delete_files""")
q("SELECT count(*) AS orders FROM booknest.orders")
spark.sql(f"CALL lake.system.rollback_to_snapshot('booknest.orders', {before})")   # undo it
q("SELECT count(*) AS orders_after_rollback FROM booknest.orders")

Output

+-------+-----------+-----+------------+-------------------------------------------------------
  -----------+
|content|file_format|month|record_count|file
  |
+-------+-----------+-----+------------+-------------------------------------------------------
  -----------+
|1      |PARQUET    |677  |4           |00000-6-518971ae-1b40-4248-96d4-9b098c173792-00001-dele
  tes.parquet|
+-------+-----------+-----+------------+-------------------------------------------------------
  -----------+
...

All four orders fell in month 677 (months since January 1970: June 2026), so one small file of (file, position) pairs replaced a rewrite of a 55 KB data file. A position delete applies only to data files with a sequence number not above its own. The count fell to 99,996, and rollback_to_snapshot restored all 100,000 orders without copying any data.

MENU