KRaft: Kafka without ZooKeeper — lessons from production

KRaft: Kafka without ZooKeeper

For years, Apache Kafka relied on ZooKeeper to manage cluster metadata and coordinate broker state. That dependency is gone. KRaft: Kafka without ZooKeeper replaces ZooKeeper's coordination with an embedded Raft-based controller quorum, eliminating an entire operational layer and unlocking faster failover, simpler deployment, and support for millions of partitions. If you're running Kafka in production, this shift fundamentally changes how you architect, scale, and troubleshoot your clusters.

What Changed — Controllers Use a Raft Quorum Instead of ZooKeeper

In traditional Kafka (pre-KRaft), every broker made requests to an external ZooKeeper ensemble. ZooKeeper held the source of truth: which brokers were alive, what partitions existed, who was the leader of each partition, and which ISR (in-sync replicas) were current. The controller—a single elected broker—watched ZooKeeper and sent commands to other brokers to execute changes. This architecture worked, but it introduced a separate system to operate and debug.

KRaft inverts this design. Kafka now embeds a Raft consensus algorithm directly in the broker layer. Instead of talking to ZooKeeper, a subset of brokers become dedicated controller quorum members (typically 3 or 5 brokers). These controllers maintain a replicated log of metadata events (topics created, partitions assigned, ISR changes) using Raft. Any broker can read this metadata log; only the active controller can write to it. When the active controller fails, Raft automatically elects a new one from the remaining quorum members.

Here's a simple comparison of the request flow:

ZooKeeper mode (legacy):
Broker → ZooKeeper (watch) → Controller reads ZooKeeper state → Controller sends command to broker

KRaft mode (modern):
Broker → Local controller node (in-memory quorum state) → Active controller writes to Raft log → All brokers read updated metadata

The metadata log itself is stored locally on each broker, making the system self-contained. There is no external dependency, no separate ensemble to provision, and no network calls to a different system for every metadata operation.

Why It Matters — Simpler Operations, Faster Failover, and More Partitions

This architectural shift delivers three major wins:

1. Simpler Operations

Running a production Kafka cluster used to require a second production-grade system: ZooKeeper. You had to provision it, secure it, monitor it, upgrade it independently, and debug two separate systems when things broke. ZooKeeper's configuration was often cryptic (tick times, minimum session timeouts, autopurge settings), and misconfiguration could cause silent election hangs or session timeouts that cascaded through your Kafka cluster.

With KRaft, you deploy Kafka brokers. That's it. No separate ZooKeeper nodes, no dual upgrade paths, no separate runbooks for ZooKeeper failures. Your monitoring and alerting tooling can focus on a single system. Your security policies (TLS, mTLS, ACLs) apply uniformly across the cluster.

2. Faster Failover

When the active KRaft controller fails, Raft detects the absence and elects a new controller in milliseconds to seconds, compared to ZooKeeper's session timeout (typically 6–30 seconds by default, often tuned higher for stability). Faster failover means faster metadata updates, which means fewer client-side errors when brokers or partitions change. In high-throughput systems processing millions of messages per second, a 10-second controller hiccup can result in thousands of failed produce or consume requests.

3. Millions of Partitions

ZooKeeper stores all metadata in memory on a single node (the leader). At scale, with millions of partitions, ZooKeeper's heap usage became the bottleneck. Many large deployments couldn't exceed a few hundred thousand partitions without hitting ZooKeeper's practical limits. KRaft distributes the metadata log across the quorum, and the actual metadata is accessed from the broker's local in-memory copy, sidestepping ZooKeeper's single-leader bottleneck. Clusters can now safely host tens of millions of partitions.

Kafka 4.0 — ZooKeeper Mode Was Removed. New Clusters Are KRaft-Only

Kafka 4.0, released in early 2024, removed ZooKeeper support entirely. You cannot create a new ZooKeeper-based cluster on 4.x. This is a hard breaking change, and for most teams, it means a migration is unavoidable if you plan to upgrade beyond Kafka 3.x.

Here's the timeline:

• Kafka 3.3 and earlier: ZooKeeper-only. KRaft available but labeled experimental.
• Kafka 3.4 and 3.5: KRaft is production-ready. ZooKeeper still supported for existing clusters.
• Kafka 3.6 and 3.7: KRaft is the default for new clusters. ZooKeeper still supported for migration.
• Kafka 4.0+: ZooKeeper removed. All new clusters must use KRaft.

If you're running Kafka 3.x on ZooKeeper today, you have a window to plan and execute your migration before upgrading to 4.x. Kafka provides tooling to migrate metadata from ZooKeeper to KRaft without downtime, but it requires coordination and testing.

Migrating — Older Clusters Migrate to KRaft Before Upgrading to 4.x

The migration path is non-trivial but well-documented. Kafka provides a migration tool to export ZooKeeper metadata and import it into a KRaft cluster. The high-level process:

1. Assess your cluster: Identify how many brokers, topics, partitions, and ACLs you have. Ensure your Kafka version is 3.4 or later (migration tooling wasn't available in earlier 3.x releases).
2. Plan quorum size: Decide how many brokers will be controller nodes. Typically 3 or 5 brokers are dedicated to the quorum; larger clusters may use 5. This quorum must be stable and reliable.
3. Enable KRaft mode on brokers: Upgrade brokers to a KRaft-capable version, then configure process.roles to include broker and optionally controller. Generate a cluster.uuid and set controller.quorum.voters with the addresses of your quorum nodes.
4. Migrate metadata: Use Kafka's migration tool to export metadata from ZooKeeper and apply it to the KRaft cluster. This includes topic configurations, ACLs, and partition assignments.
5. Switch clients: Update your producer and consumer configurations to point to the new KRaft cluster (or the same brokers, if you're doing an in-place upgrade).
6. Validate and roll back: Test thoroughly in staging. Have a rollback plan if metadata doesn't migrate cleanly.

Here's a simplified example configuration for a KRaft broker:

// broker.properties for KRaft mode
node.id=1
process.roles=broker,controller

// Unique identifier for this cluster
cluster.uuid=MkQkDj5dT2KeL5r9vPjP2Q

// Which brokers are controllers?
controller.quorum.voters=1@kafka-broker-1:9093,2@kafka-broker-2:9093,3@kafka-broker-3:9093

// Metadata storage
metadata.log.dir=/var/kafka-logs/__cluster_metadata-partition

// Network listeners
listeners=PLAINTEXT://kafka-broker-1:9092,CONTROLLER://kafka-broker-1:9093
advertised.listeners=PLAINTEXT://kafka-broker-1:9092
listener.security.protocol.map=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
controller.listener.names=CONTROLLER
inter.broker.listener.name=PLAINTEXT

// Standard broker configs still apply
log.dirs=/var/kafka-logs
broker.id=1

Notice the dual roles: broker and controller. The CONTROLLER listener handles quorum communication; the PLAINTEXT listener handles client and inter-broker traffic. Keep them separate for security and performance.

Common Mistakes

1. Undersizing the controller quorum: Using only 1 or 2 controller brokers defeats the purpose of Raft consensus. Always use an odd number (3, 5, or 7). A single controller node is a single point of failure; two nodes can't achieve quorum if one fails.

2. Mixing ZooKeeper and KRaft brokers: You cannot run ZooKeeper and KRaft modes simultaneously in the same cluster. It's an all-or-nothing migration. Don't attempt to add KRaft brokers to an existing ZooKeeper cluster or vice versa.

3. Placing all controller nodes on the same machine or network segment: The whole point of quorum consensus is fault tolerance. Distribute your controller quorum across different racks, availability zones, or machines. If all three are on the same switch and that switch fails, your cluster loses quorum.

4. Neglecting metadata log disk space: The metadata.log.dir stores the Raft log. Unlike ZooKeeper (which stores metadata in memory), this log can grow. Monitor disk usage on controller nodes. A full disk will bring down the controller and cause cluster-wide outages.

5. Forgetting to update client code: KRaft clusters expose metadata slightly differently than ZooKeeper clusters. Some legacy client libraries that made direct ZooKeeper connections (instead of using Kafka's standard APIs) will break. Ensure all your clients use the standard bootstrap.servers approach and don't rely on ZooKeeper directly.

Key Takeaways

• KRaft removes external ZooKeeper: Kafka now manages metadata via an embedded Raft quorum, simplifying the operational stack.
• Three major wins: simpler operations (no dual systems), faster failover (milliseconds instead of seconds), and millions of partitions (no single-node bottleneck).
• Kafka 4.0 removed ZooKeeper support entirely: New clusters are KRaft-only. Existing ZooKeeper clusters must migrate before upgrading to 4.x.
• Migration is tooling-supported but requires planning: Plan your quorum size, configure brokers with process.roles and controller.quorum.voters, use Kafka's migration tools, and test thoroughly.
• Use an odd-numbered quorum (3 or 5 brokers): Distribute them across fault domains. Monitor metadata log disk space and keep controller nodes stable.
• Failover is automatic: Raft handles controller election with no external coordination. Your cluster heals itself when a controller fails.