Batching and Buffering

Batching, Linger and Buffer Memory

A batch leaves when it reaches batch.size (Java 16 KB; librdkafka 1 MB) or has waited linger.ms (5 ms; Java used 0 before Kafka 4.0 129 ). When buffer.memory (32 MB) fills, Java's send() blocks; librdkafka's produce() raises BufferError.

l0654_linger.py: three linger values at 2,000 events a secondPython

from producer_lab import bench
for linger in (0, 5, 50):                             # a steady 2,000 events a second
    print(f"linger.ms={linger:<2}", bench("booknest.lab", {"linger.ms": linger},
                                        n=10_000, rate=2_000))

Output

linger.ms=0       893 msg/s     1/batch p50 5483.6 ms  p99 6272.5 ms  0 failed
linger.ms=5     1,999 msg/s     5/batch p50    5.0 ms  p99   42.1 ms  0 failed
linger.ms=50    1,999 msg/s    35/batch p50   29.0 ms  p99   58.3 ms  0 failed

Without linger every record was its own request: the producer fell behind even this rate in all six runs, and flat out it managed 32,000 events a second against 237,000 at 5 ms. Larger lingers trade latency for batch size.

MENU