One consumer group or one per instance, explained simply

When Event Volume Meets WebSocket Delivery

Building a Kafka-to-WebSocket gateway forces you to make one of the earliest architectural decisions: One consumer group or one per instance? Get this wrong and you'll either waste resources broadcasting events that nobody needs, or you'll miss critical updates because another instance already consumed them. This decision cascades into how you scale, how you handle failover, and ultimately how much infrastructure you'll need to support your real-time platform.

I've seen both approaches fail in production. A trading platform I worked on initially used a shared consumer group, which felt elegant until we added a second instance and discovered orders were reaching only one client instead of the entire room. Then we switched to per-instance groups and nearly doubled our Kafka processing cost before we optimized the fan-out logic. The difference between these two strategies boils down to a single question: do you want Kafka to distribute the work, or do you want each instance to see everything?

Shared Consumer Group: One Instance Gets Each Event

With a shared consumer group, Kafka partitions your data so that each partition is consumed by exactly one instance at a time. If you have three instances and four partitions, Kafka will assign roughly one partition per instance, and the fourth instance stays warm. When an event lands on partition 2, only the assigned instance consumes it. This minimizes duplicate processing and keeps your Kafka throughput clean.

Here's how a typical setup works. Your gateway instances all connect to Kafka as part of a group called websocket-gateway-prod. When instance A starts, it gets assigned partition 0 and 1. Instance B joins and gets partition 2. Instance C joins and gets partition 3. An event arrives on partition 1: only instance A processes it. If that event needs to reach a client connected to instance B or C, instance A must publish it somewhere else—usually Redis—so the other instances know to broadcast it.

// Shared consumer group approach
const consumer = kafka.consumer({ groupId: 'websocket-gateway-prod' });

await consumer.connect();
await consumer.subscribe({ topic: 'orders', fromBeginning: false });

await consumer.run({
  eachMessage: async ({ topic, partition, message }) => {
    const order = JSON.parse(message.value.toString());
    
    // Publish to Redis pubsub for broadcast to all instances
    await redis.publish('order-updates', JSON.stringify({
      orderId: order.id,
      status: order.status,
      timestamp: Date.now()
    }));
    
    console.log(`Instance ${process.env.INSTANCE_ID} processed order ${order.id} from partition ${partition}`);
  }
});

// Subscribe to Redis pubsub to broadcast to connected clients
redis.subscribe('order-updates', (message) => {
  const order = JSON.parse(message);
  
  // Find clients interested in this order
  const interestedClients = wsServer.clients.filter(
    client => client.orderId === order.orderId
  );
  
  interestedClients.forEach(client => {
    client.send(JSON.stringify(order));
  });
});

The advantage here is Kafka does the heavy lifting. If you have 10 million events per day and three instances, each instance processes roughly 3.3 million events. Your CPU and memory scale more predictably because the load is distributed. You also avoid re-processing the same event multiple times in your application code.

The tradeoff is complexity in your broadcast layer. Every event must flow through an intermediary—Redis, RabbitMQ, or even another Kafka topic—to reach instances that didn't consume it directly. This adds latency and creates another failure point. If Redis goes down, clients connected to instances that didn't consume the original event won't get updates.

Unique Group Per Instance: Every Instance Sees Everything

With a per-instance consumer group, each instance subscribes to all partitions of your topic. If you have three instances and four partitions, all three instances consume from all four partitions. This means every event is processed by every instance, but each instance only needs to forward it to clients connected to that instance. No intermediary required for routing—just local WebSocket delivery.

Here's what this looks like in code:

// Per-instance consumer group approach
const groupId = `websocket-gateway-${process.env.INSTANCE_ID}`;
const consumer = kafka.consumer({ groupId });

await consumer.connect();
await consumer.subscribe({ topic: 'orders', fromBeginning: false });

await consumer.run({
  eachMessage: async ({ topic, partition, message }) => {
    const order = JSON.parse(message.value.toString());
    
    // Find clients connected to THIS instance
    const interestedClients = wsServer.clients.filter(
      client => client.orderId === order.orderId
    );
    
    // Broadcast directly to connected clients, no Redis
    interestedClients.forEach(client => {
      client.send(JSON.stringify(order));
    });
    
    console.log(`Instance ${process.env.INSTANCE_ID} processed order ${order.id}, delivered to ${interestedClients.length} local clients`);
  }
});

The appeal is obvious: zero intermediary, zero extra hops, lower latency. An event reaches a client as soon as this instance processes it. You also avoid the Redis dependency—if your message broker goes down, individual instances still work for locally connected clients.

The cost is Kafka throughput. With three instances, every event is consumed three times. That's 3x the Kafka reads, 3x the CPU on your broker, and 3x the network traffic from Kafka to your instances. If you're processing 10 million events per day, you're now asking Kafka to deliver 30 million events just to three small gateways. This is wasteful for most e-commerce or SaaS platforms, but if your event volume is low—say, 100k per day—the waste is negligible and the simplicity wins.

Real-World Scenarios: When to Choose Each

Scenario 1: High-volume trading platform – You're processing 50 million order events per day. You have 10 gateway instances and 100 Kafka partitions. A shared consumer group is non-negotiable here. Each instance handles roughly 5 million events, and you route updates via Redis. Kafka throughput stays reasonable, and your instances handle only their fair share of computation.

Scenario 2: SaaS collaboration tool – You're processing 200k presence and chat events per day across 2-3 instances. Each instance averages 100-150 active users. A per-instance group is simpler. Every instance sees every event, but you're only forwarding updates to local clients. No Redis dependency, lower latency, and the Kafka "waste" is negligible at 200k daily volume.

Scenario 3: Ride-hailing network – You process 5 million driver location updates per day. You have 5 gateway instances. Your location events are partition-keyed by region. A shared group makes sense, but you must design carefully. Only instances serving that region need the update, so your Redis routing logic must be geo-aware, not just "broadcast to all instances."

Hybrid Approach: Context-Aware Routing

In some platforms, neither pure approach is optimal. I've used a hybrid strategy where you use a shared consumer group, but you add logic to skip the Redis broadcast for certain event types.

// Hybrid approach
await consumer.run({
  eachMessage: async ({ topic, partition, message }) => {
    const event = JSON.parse(message.value.toString());
    
    // For high-volume, low-relevance events (e.g., analytics), 
    // let this instance handle them and skip broadcast
    if (event.type === 'analytics') {
      // Process locally only
      await trackEvent(event);
      return;
    }
    
    // For critical events (orders, presence), broadcast via Redis
    if (event.type === 'order_status' || event.type === 'user_online') {
      await redis.publish('critical-updates', JSON.stringify(event));
      return;
    }
  }
});

This reduces Kafka waste and Redis load by filtering events at the source. Only truly broadcast-worthy events consume your intermediary. Events that don't need to reach remote clients are handled locally by the consuming instance.

Common Mistakes

• Underestimating Kafka throughput cost – I've seen teams pick per-instance groups because "it's simpler," then hit CPU limits on the broker within six months when volume doubled. Run the math: multiplying Kafka throughput by instance count is expensive at scale.
• Over-relying on Redis without redundancy – If you choose a shared group, you must make Redis highly available. I watched a platform lose client updates for two hours because Redis crashed and they had no replica. Use Redis Cluster or add a fallback publish mechanism.
• Ignoring partition skew – With shared groups, if one topic partition receives 90% of events and your Kafka cluster assigns it to a single instance, that instance becomes a bottleneck. Use proper key partitioning to distribute load evenly.
• Mixing groups accidentally – I've seen teams add a new instance without specifying the group ID correctly, creating an accidental per-instance hybrid where some instances consume everything and others consume nothing. Use environment variables for group IDs and document your topology.
• Forgetting to test failover – Both approaches can fail differently. Test what happens when an instance crashes mid-broadcast in the shared group scenario, or when an instance gets unassigned partitions during rebalancing in the per-instance scenario.

Key Takeaways

• Use a shared consumer group when event volume is high (millions per day) and you can tolerate an extra hop through Redis. Kafka becomes your load balancer.
• Use a per-instance group when event volume is low-to-medium (thousands per day) and latency is critical. Simplicity and directness win over Kafka efficiency.
• Measure your actual volume first. Don't guess. Log events for a week and calculate daily throughput, then run the math on Kafka broker CPU for both scenarios.
• If you choose shared groups, make your broadcast layer resilient. Redis crashes happen; design a fallback or use Kafka itself as a secondary broadcast channel.
• Partition your Kafka topics by intent. High-volume analytical events should go to a different topic than critical business events so you can apply different consumer group strategies per topic.
• As your platform scales, revisit this decision. What made sense at 100k events per day may not make sense at 10 million. Build monitoring to track Kafka throughput and latency so you know when to switch strategies.