Partitions as Append-Only Logs

On disk a partition is a directory, <topic>-<partition>, in each replica's log.dirs. Kafka 129 only appends to it and deletes only whole old files. Appends are sequential writes, which disks handle at full bandwidth; they go to the operating system's page cache, with durability coming from replication rather than fsync; and consumers of recent data are served from that cache, on unencrypted connections straight from file to socket (sendfile).

A partition directory: a sequence of segments, each a log file with two sparse indexes
A partition directory: a sequence of segments, each a log file with two sparse indexes

The producer builds record batches that the broker stores unchanged and consumers receive as the same bytes, so a batch is compressed once, by the producer, and decompressed only by consumers (Compression Codecs).