Kafka 129 's bin/ folder carries its own load generators: kafka-producer-perf-test.sh (Load-Testing a Producer), kafka-consumer-perf-test.sh and kafka-share-consumer-perf-test.sh for consumer and share-group throughput, kafka-e2e-latency.sh for produce-to-consume latency one record at a time (it forces the consumer's fetch wait to zero), and trogdor.sh for workloads and fault injection spread over many hosts. For cross-system comparisons, the OpenMessaging Benchmark (https://github.com/openmessaging/benchmark 451 ) framework (Apache 2.0) drives Kafka, Pulsar, RabbitMQ 28,807 and others with the same workloads; vendors' published comparisons usually use it. Whatever the tool, use your own record sizes, keys and acks (BookNest's events average 94 bytes) and repeat in alternating order. The question 6.15.4 leaves open is what reading the history costs: l0615_consume.sh reads all 390,737 events once from a topic held on local disk and once from the tiered topic with kafka-consumer-perf-test.sh --num-records 390737, three rounds:
booknest.bench 6.3122 MB/s 70112.5067 msg/s, 35.1777 MB booknest.order-history 7.7878 MB/s 86503.6529 msg/s, 35.1777 MB booknest.bench 7.4974 MB/s 83277.2805 msg/s, 35.1777 MB booknest.order-history 7.3578 MB/s 81727.0446 msg/s, 35.1777 MB booknest.bench 7.3440 MB/s 81573.4864 msg/s, 35.1777 MB booknest.order-history 7.2071 MB/s 80052.6531 msg/s, 35.1777 MB
The tiered read was as fast as the local one (0.98 to 1.23 times its rate per round), because the plug-in fetches 4 MB chunks and the "bucket" here is a local directory. Against S3, each chunk is a request with tens of milliseconds to its first byte, so old data reads slower there.