Why ZooKeeper Was Removed

Why ZooKeeper Was Removed in Kafka 4.0

Before KRaft a cluster was two distributed systems: Apache ZooKeeper stored the metadata, and one broker, elected through ZooKeeper, pushed changes to the others by RPC. KIP-500 (2019) listed the costs. A new controller had to reload all metadata from ZooKeeper, so failover on a large cluster took minutes and practical limits sat around 200,000 partitions. Brokers could drift from ZooKeeper, since changes traveled as RPCs rather than an ordered log. And every team ran, secured and upgraded a second system. In KRaft, standby controllers already hold the metadata log, brokers replay it in order, and there is one system to operate.

KRaft became production-ready in Kafka 3.3 129 (2022), 3.5 deprecated ZooKeeper mode, 3.9 is the last release that migrates a ZooKeeper cluster, and 4.0 deleted the ZooKeeper code. Tutorials that start ZooKeeper first or pass --zookeeper to the topic tool no longer apply.