Inspecting Segments

Inspecting Segments

Inspecting Segments with Low-Level Tools

kafka-dump-log.sh decodes segment files without a broker: batch headers, records (--print-data-log), indexes, and the KRaft log (--cluster-metadata-decoder). Use it when no client can answer: was this batch compressed, which leader epoch wrote it, what did the controller record?

Decoding a segment and the metadata log

# Decode a data segment and the cluster metadata log
D=/var/lib/kafka/data
seg=$D/booknest.order-events-0/00000000000000000000
dump() { docker exec $1 /opt/kafka/bin/kafka-dump-log.sh "${@:2}"; }
dump l3-b4 --files $seg.log --print-data-log | sed -n 3p |
  awk '{print $1, $2, $3, $4, $5, $6, $15, $16, $27, $28}'      # first batch header
dump l3-b4 --files $seg.log --print-data-log | sed -n 4,5p |
  awk '{print $2, $3, $4, $5, $8, $9, $14, $15, $16, $17}' | cut -c1-95   # its first records
M=$D/__cluster_metadata-0/00000000000000000000.log
dump l3-c1 --cluster-metadata-decoder --files $M |
  grep -o '"type":"[A-Z_]*"' | sort | uniq -c | sort -rn | head -6

Output

baseOffset: 0 lastOffset: 1963 count: 1964 partitionLeaderEpoch: 0 size: 215285
offset: 0 CreateTime: 1735693969000 valueSize: 110 key: 7 payload: {"event_id":12,"ts":"2025-01
offset: 1 CreateTime: 1735694179000 valueSize: 76 key: 7 payload: {"event_id":14,"ts":"2025-01-
    510 "type":"NO_OP_RECORD"
    290 "type":"PARTITION_CHANGE_RECORD"
     62 "type":"PARTITION_RECORD"
     16 "type":"BROKER_REGISTRATION_CHANGE_RECORD"
      8 "type":"CONFIG_RECORD"
      6 "type":"FEATURE_LEVEL_RECORD"

The first batch carries 1,964 uncompressed events, starting with order 7's (Topics and Partitions). The metadata log holds 62 partition creations (four topics of three plus 50 for __consumer_offsets), 290 leader and ISR changes, and no-op records the active controller appends to show it is alive. kafka-metadata-shell.sh browses a metadata snapshot as a file tree.

MENU