confluent-kafka-python

confluent-kafka-python and librdkafka

confluent-kafka-python (https://github.com/confluentinc/confluent-kafka-python 515 ) (Apache 2.0, 2.15.1 of 10 September 2026) is a thin C extension over librdkafka (https://github.com/confluentinc/librdkafka 1,060 ), the C client Magnus Edenhill started in 2012 (BSD 2-clause), now maintained by Confluent 17,245 ; the wheel has it compiled in. The same library sits under kcat (kcat), the Go and .NET clients and one Node.js 2,131 client below, so they all share Java's property names (linger.ms) and librdkafka's own defaults.

Python calls never touch the network; librdkafka's native threads do, and Linux can list them:

clients/l06121_threads.py: the threads behind a producer and a consumerPython

"""l06121_threads.py: the native threads behind a confluent-kafka producer and consumer."""
import os, collections, confluent_kafka as ck
def threads():                                    # Linux names each native thread
    tasks = os.listdir("/proc/self/task")
    return dict(collections.Counter(open(f"/proc/self/task/{t}/comm").read().strip()
                                    for t in tasks))
print("confluent-kafka", ck.__version__, "on librdkafka", ck.libversion()[0])
producer = ck.Producer({"bootstrap.servers": "localhost:32092"})
producer.list_topics(timeout=5)                   # connect, fetch metadata
print("producer:", threads())
consumer = ck.Consumer({"bootstrap.servers": "localhost:32092",
                        "group.id": "booknest-threads"})
consumer.subscribe(["booknest.order-events"])
consumer.poll(5)                                  # join the group, fetch
print("+ consumer:", threads())

Output

confluent-kafka 2.15.1 on librdkafka 2.15.1
producer: {'python': 1, 'rdk:main': 1, 'rdk:broker-1': 1, 'rdk:broker1': 1}
+ consumer: {'python': 1, 'rdk:main': 2, 'rdk:broker-1': 3, 'rdk:broker1': 2}

Each client owns a main thread and one thread per broker connection: rdk:broker1 talks to node 1, and rdk:broker-1 is a logical broker with no node ID, first the bootstrap address and, in the consumer, the group coordinator. Callbacks wait in a queue until you call poll(), consume() or flush(), and threads do not survive fork(): create clients inside each worker process.

MENU