Idempotent Producers

A retry duplicates a record when the broker wrote it but the answer was lost. An idempotent producer gets a producer ID (PID) and numbers its batches; the leader drops repeats. Java producers are idempotent by default since Kafka 3.0 129 ; librdkafka's are not.

Which producers write a producer ID and sequence numbers

PY=/home/dev/v7-l3/kafka-venv/bin/python
K="docker exec -i l3-kafka /opt/kafka/bin"
B="--bootstrap-server l3-kafka:9092"
$K/kafka-topics.sh $B --create --topic booknest.idem --partitions 1 >/dev/null 2>&1
for idem in false true; do
  $PY -c "from confluent_kafka import Producer
p = Producer({'bootstrap.servers': 'localhost:33092', 'enable.idempotence': '$idem'})
p.produce('booknest.idem', '{\"order_id\": 7}', '7'); p.flush()"
done
echo '{"order_id": 7}' | $K/kafka-console-producer.sh $B --topic booknest.idem
$K/kafka-dump-log.sh --files /var/lib/kafka/data/booknest.idem-0/00000000000000000000.log |
  grep -o 'baseOffset: [0-9]*\|producerId: [-0-9]*\|baseSequence: [-0-9]*' | paste - - -

Output

baseOffset: 0   baseSequence: -1   producerId: -1
baseOffset: 1   baseSequence: 0   producerId: 2
baseOffset: 2   baseSequence: 0   producerId: 3

The default Python producer wrote no PID; the idempotent one got PID 2, the console producer PID 3 unasked. A restarted producer gets a new PID: surviving crashes needs transactions (Transactions and Exactly-Once).

MENU