Java hashes a key with murmur2 modulo the partition count, and fills one partition's batch with keyless records before moving on (sticky, KIP-480; since 3.3 it favors fast brokers, KIP-794). librdkafka hashes keys with CRC32 by default.
from confluent_kafka import Producer
def partitions(config, keys): # partition of each record, in send order
p, seen = Producer({"bootstrap.servers": "localhost:33092", **config}), {}
for i, key in enumerate(keys):
p.produce("booknest.order-events", str(i), key,
on_delivery=lambda e, m: seen.update({int(m.value()): m.partition()}))
p.flush()
return [seen[i] for i in range(len(keys))]
keys = [str(k) for k in range(1, 13)]
print("keys 1-12, default:", *partitions({}, keys))
print("keys 1-12, murmur2:", *partitions({"partitioner": "murmur2_random"}, keys))
seq = partitions({"linger.ms": 50}, [None] * 1000)
switches = sum(a != b for a, b in zip(seq, seq[1:]))
print(f"1,000 keyless: {switches} partition switches")Output
keys 1-12, default: 2 1 1 1 1 1 0 2 0 0 0 0 keys 1-12, murmur2: 0 2 2 1 0 1 0 0 2 1 0 1 1,000 keyless: 92 partition switches
The murmur2 line matches Kafka 129 's Java console producer for the same keys. The keyless records switched partition far less often than the 667 times random choice would give, and twelve keyless lines from the Java console producer all went to one partition. Give all clients of a keyed topic one partitioner.