Why Queues (KIP-932)

Why Kafka Needed a Queue Semantic (KIP-932)

Consumer groups are built for ordered streams, which makes them awkward work queues. Parallelism stops at the partition count, one offset per partition means one failing record blocks those behind it, and "retry this one later" needs home-made retry topics. Teams therefore ran RabbitMQ 28,807 or Amazon SQS next to Kafka 129 for jobs such as e-mails or packing orders.

KIP-932, Queues for Kafka, adds share groups on the same topics: any number of consumers read the same partitions, the broker hands each record to one consumer at a time under a time-limited acquisition lock, counts deliveries, and tracks acknowledgments per record. Ordering is given up, which suits independent jobs.