The core pattern is consume-transform-produce: read a batch, write the results, and commit the input offsets inside the same transaction, so that results and progress move together. After a crash the job resumes from the last committed offsets, and the results of the unfinished batch are aborted.
| Situation | Pattern |
|---|---|
| Kafka 129 to Kafka, your own code | Transactions with send_offsets_to_transaction() |
| Kafka Streams 129 | processing.guarantee=exactly_once_v2 (Kafka Streams) |
| Flink 129 to Kafka | Two-phase commit sink (ksqlDB and Apache Flink) |
| Kafka to a database | Idempotent upsert keyed by event_id, or offsets in the same database transaction |
| Database to Kafka | Transactional outbox table plus CDC (Orchestration and Pipelines) |
Kafka's transactions stop at Kafka's edge: an e-mail sent or a card charged inside the loop happens again when the batch is retried. Make such side effects idempotent, or move them to a consumer that records what it has done.