Orphan Files and Manifests

Removing Orphan Files and Rewriting Manifests

Orphan files are files in the table's location that no metadata references, left by failed jobs and lost commit races; they only cost storage. remove_orphan_files lists the table's location and deletes what metadata does not mention. The script plants a stray file from a "crashed" writer, then also rewrites the 13 manifests that compaction left behind:

cleanup.py: delete an orphan file, rewrite manifests, drop the scratch tablePython

"""Orphan files and manifests: remove a stray file, then rewrite the manifests."""
import subprocess, time
from lake import spark
S = "booknest.orders_stream"
good = spark.sql(f"SELECT file_path FROM {S}.files LIMIT 1").first()[0]
stray = good.rsplit("/", 1)[0] + "/00000-9-crashed-writer.parquet"     # left by a failed job
subprocess.run(["mc", "cp", "-q"] + [p.replace("s3://", "lake/") for p in (good, stray)],
               check=True, capture_output=True)    # copy through mc's alias for MinIO
try:
    spark.sql(f"CALL lake.system.remove_orphan_files(table => '{S}', older_than => now())")
except Exception as e:
    print("procedure refused:", str(e).split(".")[0])
jvm = spark._jvm                                    # the same action through the Java API
table = jvm.org.apache.iceberg.spark.Spark3Util.loadIcebergTable(spark._jsparkSession, S)
res = (jvm.org.apache.iceberg.spark.actions.SparkActions.get().deleteOrphanFiles(table)
       .olderThan(int(time.time() * 1000)).usePrefixListing(True).execute())
for path in res.orphanFileLocations():
    print("deleted orphan:", path.split("/orders_stream/")[1])
r = spark.sql(f"CALL lake.system.rewrite_manifests('{S}')").first()
print(f"rewrite_manifests: {r.rewritten_manifests_count} -> {r.added_manifests_count}")
spark.sql(f"DROP TABLE {S} PURGE")                  # the scratch table has served its purpose

Output

procedure refused: Cannot remove orphan files with an interval less than 24 hours
deleted orphan: data/order_ts_day=2026-06-27/00000-9-crashed-writer.parquet
rewrite_manifests: 13 -> 1

The refusal is a safety rule worth keeping: a file written a minute ago may belong to a commit still in flight, and deleting it would corrupt that commit, so the procedure insists on a 24-hour margin (three days by default). The Java action has no such guard. rewrite_manifests merged 13 manifests into one sorted by partition, so planning reads one file, not thirteen. Run orphan removal weekly with the default margin.

MENU