Apache Pulsar

Apache Pulsar 129 (https://github.com/apache/pulsar 15,340 ) (Apache 2.0), born at Yahoo and a top-level Apache project since 2018, splits serving from storage. Its brokers are stateless; the data lives in Apache BookKeeper, a replicated log store whose nodes are bookies. A topic is a chain of ledgers, each striped across several bookies, so a failed broker's topics move to another at once with nothing to copy, and new bookies take new ledgers without a reassignment. Metadata lives in Oxia (recommended), ZooKeeper, or RocksDB on one node.

Kafka 129 keeps each partition on its brokers; Pulsar separates brokers from storage

Pulsar builds in tenants and namespaces with their own policies, geo-replication, tiered storage, and four subscription types on any topic: Exclusive (the default), Failover, Shared (a round-robin queue) and Key_Shared (a queue that keeps each key on one consumer), plus negative acknowledgment, delayed delivery and dead-letter topics. Unlike Kafka, it deletes a message once every subscription has acknowledged it, unless the namespace sets a retention policy. Version 4.2.4 (3 Aug 2026) is current, 4.0 is the long-term-support line, and 5.0.0-M2 previews 5.0. Here apachepulsar/pulsar:4.2.4 runs bin/pulsar standalone (broker, bookie and RocksDB in one JVM) as l1-pulsar on ports 31650 (clients) and 31080 (REST admin).

The Python client (pip 21,050 install pulsar-client, 3.13.0) writes the events keyed by order to one unpartitioned topic; four packers share a Key_Shared subscription, and analytics is never read.

pulsar_orders.py: a Key_Shared queue and an unread subscription on one topicPython

"""pulsar_orders.py: 20,000 BookNest events keyed by order, four Key_Shared packers."""
import json
from collections import Counter, defaultdict
from concurrent.futures import ThreadPoolExecutor
from urllib.request import urlopen
import pulsar
TOPIC = "persistent://public/default/booknest-order-events"     # created on first use
client = pulsar.Client("pulsar://localhost:31650",
                       logger=pulsar.ConsoleLogger(pulsar.LoggerLevel.Error))
client.subscribe(TOPIC, "analytics")                # Exclusive (the default), never read here
packers = [client.subscribe(TOPIC, "packers", consumer_type=pulsar.ConsumerType.KeyShared,
                            consumer_name=f"packer-{i}") for i in range(1, 5)]
producer = client.create_producer(TOPIC, batching_type=pulsar.BatchingType.KeyBased,
                                  block_if_queue_full=True)    # wait, never drop, when busy
with open("data/order_events.jsonl") as f:
    for line, _ in zip(f, range(20_000)):
        key = str(json.loads(line)["order_id"])
        producer.send_async(line.encode(), None, partition_key=key)
producer.flush()
seen = defaultdict(list)                             # order_id -> [(packer, event_id), ...]
def pack(consumer):
    try:
        while msg := consumer.receive(timeout_millis=3000):
            e = json.loads(msg.data())
            seen[e["order_id"]].append((consumer.consumer_name(), e["event_id"]))
            consumer.acknowledge(msg)
    except pulsar.Timeout:
        pass
with ThreadPoolExecutor() as pool:                   # one thread per packer
    list(pool.map(pack, packers))
print(dict(sorted(Counter(p for evs in seen.values() for p, _ in evs).items())))
split = sum(len({p for p, _ in evs}) > 1 for evs in seen.values())
disorder = sum([e for _, e in evs] != sorted(e for _, e in evs) for evs in seen.values())
print(f"orders {len(seen):,}, split across packers {split}, out of order {disorder}")
stats = json.load(urlopen("http://localhost:31080/admin/v2/persistent/public/default/"
                          "booknest-order-events/stats"))
print("backlog:", {s: v["msgBacklog"] for s, v in stats["subscriptions"].items()})
client.close()

Output

{'packer-1': 4647, 'packer-2': 5551, 'packer-3': 5149, 'packer-4': 4653}
orders 5,487, split across packers 0, out of order 0
backlog: {'analytics': 20000, 'packers': 0}

Four consumers shared one unpartitioned topic, yet no order's events were split or reordered: share-group parallelism with consumer-group ordering (Consumer or Share Groups?). Each subscription is its own cursor, so analytics still holds all 20,000 events. The price is weight: the standalone idled at 720-780 MiB against 286 MiB for a Kafka broker (Docker Images), and production has three tiers to run. StreamNative's managed Pulsar, which also speaks the Kafka protocol, is in StreamNative.

MENU