Retention and Compaction

Retention by Time, Size and Log Compaction

With cleanup.policy=delete (the default) the broker removes closed segments whose newest record is older than retention.ms (7 days), or the oldest ones once a partition exceeds retention.bytes (unlimited by default). With compact, the log cleaner rewrites closed segments keeping only each key's latest record; a null value (a tombstone) removes the key after delete.retention.ms. This cluster checks retention every 10 seconds instead of 5 minutes (log.retention.check.interval.ms).

The order events under delete and compact cleanup policies

# The same events under two cleanup policies: delete after 7 days, or keep the latest per key
K="docker exec l3-c1 /opt/kafka/bin"
B="--bootstrap-server l3-b4:9092"
$K/kafka-topics.sh $B --create --topic booknest.order-events-7d --config retention.ms=604800000
$K/kafka-topics.sh $B --create --topic booknest.order-status --config cleanup.policy=compact \
  --config min.cleanable.dirty.ratio=0.01
for t in booknest.order-events-7d booknest.order-status; do
  /home/dev/v7-l3/kafka-venv/bin/python listings/load_events.py $t >/dev/null
done
sleep 60                                                  # let retention and the cleaner run
for t in -2 -1; do                                        # log start and log end offsets
  echo "7d --time $t: $($K/kafka-get-offsets.sh $B --topic booknest.order-events-7d \
    --time $t | sort | cut -d: -f3 | xargs)"
done
$K/kafka-console-consumer.sh $B --topic booknest.order-status --from-beginning \
  --timeout-ms 10000 --formatter-property print.key=true 2>/dev/null > status.txt
echo "records left in booknest.order-status: $(wc -l < status.txt)"
awk -F'\t' '$1 == 7' status.txt                           # everything left for order 7

Output

Created topic booknest.order-events-7d.
Created topic booknest.order-status.
7d --time -2: 128971 131080 130686
7d --time -1: 128971 131080 130686
records left in booknest.order-status: 102049
7   {"event_id":4277,"ts":"2025-01-09T00:16:19Z","type":"order_delivered","order_id":7}

The seven-day topic emptied within a minute (log start equals log end) because retention compares record timestamps with the clock, and every event is months old; hence retention.ms=-1 on the main topic. The compacted topic kept 102,049 records, about one per order plus segments not yet cleaned, and order 7 is reduced to order_delivered. Compaction suits latest-value data such as customers and the catalog; consumers must tolerate the offset gaps it leaves.

MENU