Throughput and Latency

Broker Configuration for Throughput and Latency

A broker's request path has two thread pools: network threads read requests from sockets and write responses, I/O threads append to the log and serve fetches, and a bounded queue sits between them. Writes go to the operating system's page cache, not straight to disk, and Kafka 129 never calls fsync by default; durability comes from replicas (Replication). Reads of recent data come from the same cache, and plaintext fetches are sent with zero-copy (sendfile), from page cache to socket without entering the JVM, which TLS prevents (Encryption in Transit with TLS). The settings that govern all this, as the broker reports them:

listings/l0615_broker.sh: performance settings, as the running broker reports themShell
K='num.network.threads|num.io.threads|num.replica.fetchers|queued.max.requests'
K+='|socket.send.buffer.bytes|socket.receive.buffer.bytes|log.flush.interval.messages'
K+='|log.segment.bytes|message.max.bytes'
docker exec l3-kafka-ts /opt/kafka/bin/kafka-configs.sh --bootstrap-server l3-kafka-ts:9092 \
  --describe --all --entity-type brokers --entity-name 1 | sed 's/ sensitive=.*//' |
  grep -E "^  ($K)="
Output
  log.flush.interval.messages=9223372036854775807
  log.segment.bytes=1073741824
  message.max.bytes=1048588
  num.io.threads=8
  num.network.threads=3
  num.replica.fetchers=1
  queued.max.requests=500
  socket.receive.buffer.bytes=102400
  socket.send.buffer.bytes=102400

All are defaults. Raise num.network.threads and num.io.threads only when metrics say so: the NetworkProcessorAvgIdlePercent and RequestHandlerAvgIdlePercent gauges (Monitoring and Kafka UIs) falling below about 30% mean the pools are busy. The flush interval of Long.MAX_VALUE means "let the OS decide" and should stay so. These thread settings are dynamic: kafka-configs.sh --alter --entity-type brokers --entity-default changes them on a running cluster without a restart.

The bigger wins are below Kafka: leave most memory to the page cache (a 4-6 GB heap suits even large brokers), use XFS or ext4 on SSDs, set vm.swappiness=1, and raise the open-file limit and vm.max_map_count, since every segment and index is a file and indexes are memory-mapped.