Streams as Table Commits

Landing Streams as Table Commits, Not Files

Consumers that drop files into a folder give readers half-written files, duplicates after restarts and a swamp of tiny files (Data Lakes and Swamps). A lakehouse sink turns the stream into table commits instead: writers produce Parquet 129 files nobody can see yet, and at each interval one atomic commit adds them with the offsets they cover.

The Iceberg sink: tasks write files, one coordinator commits them with their offsets
The Iceberg 129 sink: tasks write files, one coordinator commits them with their offsets

A crash before a commit leaves only unreferenced files (Orphan Files and Manifests) and uncommitted offsets to re-read. The interval is the knob: shorter means fresher data and more, smaller files.