Topic Configuration

Topic Configuration

Partitions, Replication Factor and Topic Configuration

The partition count caps a topic's consumer parallelism, the replication factor sets its copies (three in production), and configs override broker defaults for that topic. kafka-configs.sh changes them on a live topic, and --describe --all shows where each value comes from, most specific source first.

Overriding, inspecting and removing a topic config

# Override a topic setting, see where each value comes from, then remove the override
K="docker exec -i l3-kafka /opt/kafka/bin"
B="--bootstrap-server l3-kafka:9092"
E="--entity-type topics --entity-name booknest.order-events"
$K/kafka-configs.sh $B $E --alter --add-config max.message.bytes=262144 >/dev/null
$K/kafka-configs.sh $B $E --describe --all |
  grep -E ' (max.message.bytes|cleanup.policy|min.insync.replicas)=' |
  sed -E 's/ sensitive=false synonyms=\{/  from /; s/:[^,}]*//g; s/\}//; s/,/ >/g'
$K/kafka-configs.sh $B $E --alter --delete-config max.message.bytes >/dev/null

Output

  cleanup.policy=delete  from DEFAULT_CONFIG
  max.message.bytes=262144  from DYNAMIC_TOPIC_CONFIG > DEFAULT_CONFIG
  min.insync.replicas=1  from DYNAMIC_DEFAULT_BROKER_CONFIG > DEFAULT_CONFIG

Size partitions for peak throughput over one consumer's rate, with headroom. A replication factor above the broker count fails (InvalidReplicationFactorException here), and changing it later is a reassignment (Reassigning Partitions). Beware one-replica topics: eligible leader replicas (KIP-966) cap the effective minimum ISR at the replication factor, so such a topic accepts acks=all writes despite min.insync.replicas=2, and loses them with its disk.

MENU