Reassigning Partitions

kafka-reassign-partitions and Cluster Maintenance

Kafka 129 never moves replicas by itself. kafka-reassign-partitions.sh executes a JSON plan of new replicas per partition (--generate proposes one) with throttled copying; --verify reports progress and lifts the throttle. On the six-node cluster a one-replica topic gets three replicas; then broker 6 is cordoned (Kafka 4.3, KIP-1066).

Raising a topic's replication factor and cordoning a broker

# Six-node cluster: give a one-replica topic three replicas, then cordon broker 6
K="docker exec -i l3-c1 /opt/kafka/bin"
B="--bootstrap-server l3-b4:9092"
$K/kafka-topics.sh $B --create --topic booknest.customers --replication-factor 1 \
  --config cleanup.policy=compact >/dev/null 2>&1           # the mistake: one copy
cat > move.json <<'J'
{"version": 1, "partitions": [
  {"topic": "booknest.customers", "partition": 0, "replicas": [4, 5, 6]},
  {"topic": "booknest.customers", "partition": 1, "replicas": [5, 6, 4]},
  {"topic": "booknest.customers", "partition": 2, "replicas": [6, 4, 5]}]}
J
R="$K/kafka-reassign-partitions.sh $B --reassignment-json-file /dev/stdin"
$R --execute --throttle 5000000 < move.json >/dev/null      # copy at most 5 MB/s
until $R --verify < move.json | grep -q 'Clearing topic'; do sleep 2; done  # lifts throttle
$K/kafka-topics.sh $B --describe --topic booknest.customers | cut -f3-6 | tail -3 | tr '\t' ' '
$K/kafka-configs.sh $B --entity-type brokers --entity-name 6 --alter \
  --add-config 'cordoned.log.dirs=*' >/dev/null             # no new replicas on broker 6
$K/kafka-topics.sh $B --create --topic booknest.reviews --replication-factor 2 >/dev/null 2>&1
$K/kafka-topics.sh $B --describe --topic booknest.reviews | cut -f5 | tail -3 | tr '\n' ' '

Output

Partition: 0 Leader: 4 Replicas: 4,5,6 Isr: 4,5,6
Partition: 1 Leader: 5 Replicas: 5,6,4 Isr: 4,5,6
Partition: 2 Leader: 6 Replicas: 6,4,5 Isr: 4,5,6
Replicas: 4,5 Replicas: 5,4 Replicas: 5,4

The empty topic's new replicas joined the ISR at once; with data they first copy at the throttled rate. Each leader stayed on the broker with the old copy, here also the preferred leader (listed first); otherwise the controller's five-minute check or kafka-leader-election.sh moves leadership there. A cordoned broker keeps its partitions but gets no new ones, so the next topic avoided broker 6. To retire a broker, cordon it, reassign its partitions away, stop it and run kafka-cluster.sh unregister.

MENU