Consumer groups and rebalancing: lessons from production

Consumer groups and rebalancing: The foundation of scalable Kafka systems

When you build a real-time system with Kafka, you'll quickly discover that a single consumer can't keep up with high-volume topics. That's where consumer groups come in—they let you scale horizontally by distributing partitions across multiple consumers. But this distribution isn't magic, and it comes with trade-offs you need to understand. Today we're diving into how Kafka spreads work across consumers, what happens when things change, and how to minimize the pain when they do.

One partition, one consumer — The fundamental rule

Kafka enforces a simple contract: within a consumer group, each partition is assigned to exactly one consumer. This is the guarantee that prevents duplicate processing and maintains ordering per partition. If you have a topic with six partitions and you create a consumer group with three consumers, Kafka will assign two partitions to each consumer automatically.

Let me walk you through a concrete scenario. Suppose you're processing order events from an e-commerce platform. Your topic orders has six partitions—one for each region your business operates in. You start with two consumer instances in your group, maybe running on different servers. Kafka assigns partitions 0, 1, and 2 to consumer A, and partitions 3, 4, and 5 to consumer B. Each consumer reads independently, maintains its own offset, and processes messages in order within its assigned partitions.

This design is elegant because it eliminates the need for distributed locks or global coordination. Consumer A never competes with Consumer B for partition 2—it owns it completely. Ordering is preserved per partition, which is usually the granularity that matters for business logic (e.g., all events for a given customer, region, or order ID tend to hash to the same partition).

Here's how you'd typically set up a consumer group in code:

const kafka = new Kafka({
  clientId: 'order-processor',
  brokers: ['kafka-1:9092', 'kafka-2:9092'],
});

const consumer = kafka.consumer({
  groupId: 'order-processing-group', // All instances share this ID
});

await consumer.subscribe({ topic: 'orders', fromBeginning: false });

await consumer.run({
  eachMessage: async ({ topic, partition, message }) => {
    console.log(`Processing order from partition ${partition}`);
    // Your business logic here
    await processOrder(message);
  },
});

Every instance that starts with the same groupId becomes part of the group. Kafka automatically partitions the work.

Extra consumers sit idle — Understanding partition limits

Here's a reality that catches people off guard: if you have more consumers than partitions, some consumers will sit idle. This is a hard constraint. Kafka can't split a single partition across consumers (that would break ordering and require distributed state), and it won't assign the same partition to multiple consumers in the same group.

Imagine your team decided to scale aggressively. You've got six partitions in your orders topic, but you deploy eight consumer instances expecting to handle more throughput. When those eight instances join the group, Kafka assigns one partition to each of the first six consumers. The remaining two consumers are assigned nothing—they're stuck in standby mode, consuming CPU and memory but doing no work.

This is wasteful, but it's also a safety mechanism. Those idle consumers are ready to take over if an active consumer crashes. It's a form of passive redundancy. However, from a cost perspective, you're burning resources unnecessarily.

The practical lesson: your partition count should match or slightly exceed your expected consumer count. If you anticipate needing 10 concurrent consumers to handle your peak load, design your topic with at least 10 partitions. If your partition count is fixed (perhaps for other reasons), don't deploy more consumers than you have partitions unless you have a very good reason to accept idle replicas.

Let's say you check the current assignment in your group:

const admin = kafka.admin();
await admin.connect();

const groups = await admin.describeGroups(['order-processing-group']);
const group = groups.groups[0];

group.members.forEach((member) => {
  console.log(`Member: ${member.memberId}, topics: ${member.memberMetadata.topics}`);
});

await admin.disconnect();

You'd see something like: eight members, but only six of them have partition assignments. The other two have empty lists.

Rebalances pause work — The cost of change

Every time a consumer joins or leaves the group—whether by design, crash, or network hiccup—Kafka triggers a rebalance. During a rebalance, all consumers in the group pause processing, the coordinator revokes their current assignments, and then reassigns partitions from scratch. It's a "stop the world" moment for your consumer group.

In production, I've seen rebalances cause 5–30 second pauses depending on the group size, network latency, and broker load. For a ride-hailing platform processing location updates in real-time, that's noticeable. For a batch job processing daily reconciliation, it barely matters. But you need to know it's happening.

Here's what happens step-by-step:

1. A new consumer instance starts and sends a join request to the broker.
2. The broker (acting as group coordinator) stops accepting messages from existing consumers and waits for all members to acknowledge the rebalance.
3. Kafka revokes the old partition assignments from each consumer.
4. Kafka computes new assignments (more on strategies below).
5. Kafka sends the new assignments to each consumer.
6. Consumers restart from their stored offsets on their newly assigned partitions.

During steps 2–6, no messages are being processed from that topic in that consumer group. If you have a downstream system depending on low-latency delivery, this gap is a problem.

A real-world example: you're running a CRM system that syncs customer data from Kafka. You have a consumer group with four instances. One crashes, and Kafka immediately triggers a rebalance. For 10 seconds, the sync stops. Customer support sees stale data. Bad day.

You can catch and log rebalance events:

await consumer.run({
  eachMessage: async ({ topic, partition, message }) => {
    await processOrder(message);
  },
  eachPartition: async ({
    topic,
    partition,
    heartbeat,
    isRunning,
    isPartitionEof,
  }) => {
    // This is called as partitions are assigned/revoked
  },
});

consumer.on(consumer.events.GROUP_JOIN, async (payload) => {
  console.log('Rebalance triggered, rejoining group', payload);
});

consumer.on(consumer.events.CRASH, async (payload) => {
  console.error('Consumer crashed', payload);
});

Reduce rebalance pain — Sticky and static strategies

Kafka provides two main strategies to limit rebalance disruption: cooperative sticky assignment and static membership.

Cooperative sticky assignment

The default rebalancing strategy was "eager"—revoke everything, reassign everything. Newer versions of Kafka support "cooperative" mode, which tries to keep existing assignments stable and only moves partitions when necessary.

In sticky assignment, if consumer A is assigned partitions [0, 1] and consumer B is assigned [2, 3], and consumer C joins, the coordinator tries to give C one partition while keeping A and B mostly intact. This reduces the number of consumers that pause simultaneously.

const consumer = kafka.consumer({
  groupId: 'order-processing-group',
  sessionTimeout: 30000,           // Default 10 seconds
  heartbeatInterval: 10000,         // Default 3 seconds
  rebalanceTimeout: 60000,          // Default 60 seconds
  allowAutoTopicCreation: false,
  partitionAssigners: [
    PartitionAssigners.cooperative, // Sticky and cooperative
  ],
});

This is better, but not perfect. If you need truly minimal downtime, static membership is the answer.

Static membership

Static membership lets you assign a permanent ID to each consumer instance. If that instance crashes and restarts with the same ID within a timeout window, Kafka assumes it's the same consumer and doesn't trigger a rebalance. The partition stays "held" for it.

In a Kubernetes environment, you'd use a StatefulSet so each pod gets a stable identity:

const consumer = kafka.consumer({
  groupId: 'order-processing-group',
  sessionTimeout: 300000,           // 5 minutes — gives time for restart
  groupInstanceId: `order-processor-${process.env.POD_NAME}`, // Stable ID
});

Now if your pod restarts, Kafka holds its partitions for 5 minutes. The instant the pod comes back and rejoins with the same instance ID, it gets its old partitions back. Zero rebalance. This is game-changing for latency-sensitive systems.

Trade-off: if a consumer dies for longer than the session timeout, you're forced to rebalance anyway. But in practice, 5-minute timeouts are generous enough for most infrastructure restarts.

Common mistakes

• Ignoring partition count relative to consumer count. Deploying 20 consumers for a 5-partition topic wastes resources. Right-size your partitions during topic creation, or repartition early when you see the problem.
• Setting session timeouts too aggressively. A 5-second timeout causes rebalances on every network blip. I've seen 30–60 second timeouts work better for most workloads.
• Not using static membership in production. If you care about latency (and you probably do), enable it. The downside is minimal.
• Forgetting to commit offsets safely. During a rebalance, offsets must be committed before the revocation completes. Use async commits with a flush, or use manual offset management if you need strict control.
• Not monitoring rebalance frequency. If you see rebalances every 30 seconds, something's wrong—maybe a consumer is crashing repeatedly, or your max.poll.interval is too short. Instrument it.

Key takeaways

• Each partition within a consumer group is assigned to exactly one consumer; this preserves ordering and eliminates locks.
• More consumers than partitions = idle consumers; design your partition count to match expected concurrency.
• Rebalances pause all processing in a group; they're unavoidable but can be minimized with cooperative sticky assignment or static membership.
• Static membership with a stable instance ID and a generous session timeout is the best defence against rebalance storms in production.
• Monitor rebalance frequency and investigate spikes; they're a canary for crashes, network issues, or configuration problems.