Competing consumers: a practical Design Patterns guide

Competing Consumers: Scale Your Workers Without Losing Messages

You've hit the ceiling on a single worker processing your message queue, and adding more doesn't seem to help—in fact, it sometimes makes things worse. The competing consumers pattern solves this by letting multiple workers read from the same queue, each claiming a message so no duplicate processing happens. This is how platforms handling millions of transactions stay responsive at 3 AM when load spikes unexpectedly.

Many Workers Reading from the Same Queue

The core idea is straightforward: instead of one worker pulling messages from a queue, you deploy many. Each worker competes for the next available message, grabs it, processes it, and acknowledges completion. When a message is claimed by one worker, the queue ensures no other worker sees it. This is your horizontal scaling mechanism without the complexity of partitioning or sharding logic in your application.

I've used this pattern extensively in ride-hailing platforms. We had a queue for driver location updates—thousands arriving per second. One worker couldn't keep up. With five competing workers, we handled the load effortlessly. With ten, we hit database contention. The pattern doesn't magically solve all problems, but it distributes work intelligently.

Here's a practical setup using NestJS with a message queue like RabbitMQ:

import { Injectable } from '@nestjs/common';
import * as amqp from 'amqplib';

@Injectable()
export class DriverLocationConsumer {
  private connection: amqp.Connection;
  private channel: amqp.Channel;

  async connect() {
    this.connection = await amqp.connect('amqp://localhost');
    this.channel = await this.connection.createChannel();
    
    // Prefetch: only one message per worker at a time
    await this.channel.prefetch(1);
    
    const queue = 'driver-locations';
    await this.channel.assertQueue(queue, { durable: true });
    
    // Multiple instances of this consumer compete for messages
    await this.channel.consume(queue, async (msg) => {
      if (msg) {
        try {
          const location = JSON.parse(msg.content.toString());
          await this.processDriverLocation(location);
          this.channel.ack(msg); // Only ack after success
        } catch (error) {
          console.error('Failed to process location:', error);
          this.channel.nack(msg, false, true); // Requeue on failure
        }
      }
    });
  }

  private async processDriverLocation(location: any) {
    // Update driver position in database
    // Notify nearby riders
    console.log(`Processed location for driver ${location.driverId}`);
  }
}

Notice the prefetch(1) call—this is critical. It tells the queue to give each worker only one message at a time. Without it, one fast worker might grab ten messages while slower workers sit idle, defeating the purpose of competing consumers.

Each Message Is Processed Once

The guarantee that each message gets processed exactly once is what separates competing consumers from naive multi-threading. When a worker claims a message, it sends an acknowledgment back to the queue only after processing completes successfully. If the worker crashes before acknowledging, the message goes back to the queue for another worker to retry.

This requires your queue system to support acknowledgments. RabbitMQ, Amazon SQS, Kafka (with consumer groups), and Google Cloud Pub/Sub all do. Your application must also be disciplined: only acknowledge after you've persisted results somewhere durable.

In a trading system I worked on, we processed buy/sell orders from a queue. A worker crashed mid-process, leaving half of an order executed. Because we hadn't acknowledged the message yet, the queue redelivered it. The new worker saw the same order, checked our database, realized half of it was done, and completed it correctly. Without that acknowledgment discipline, we'd have lost money.

async processOrder(order: Order) {
  // DO NOT ack yet
  
  // Step 1: Reserve funds
  const reservation = await this.reserveFunds(order.userId, order.amount);
  
  // Step 2: Execute trade
  const execution = await this.executeTrade(order);
  
  // Step 3: Write to database (durable storage)
  await this.database.orders.create({
    id: order.id,
    status: 'completed',
    reservationId: reservation.id,
    executionId: execution.id,
  });
  
  // ONLY NOW ack the message
  // If anything above fails, the order message stays in queue
}

The key insight: your queue can't guarantee exactly-once semantics on its own. Your code has to design for it by making operations idempotent and checking state before acting.

Scale Horizontally with Load

Horizontal scaling with competing consumers is nearly automatic. When your monitoring shows queue depth increasing, you spin up another worker instance. It connects to the same queue, starts competing for messages, and immediately helps reduce backlog. No configuration changes needed. No rebalancing logic. Just deploy another container.

I tracked this in a travel booking system processing accommodation reservations. At 10 AM every Monday (business travelers planning trips), queue depth spiked. We went from 3 workers baseline to 8 workers during peak hours, then back down. The autoscaler watched queue depth and added/removed workers dynamically. Cost savings were real—we only paid for extra capacity when needed.

Here's a conceptual setup with Kubernetes:

// Deployment manifest for competing consumer
apiVersion: apps/v1
kind: Deployment
metadata:
  name: order-processor
spec:
  replicas: 3  // Start with 3, HPA will scale this
  template:
    spec:
      containers:
      - name: worker
        image: my-order-processor:latest
        env:
        - name: QUEUE_HOST
          value: rabbitmq-service
        - name: CONSUMER_CONCURRENCY
          value: "1"  // One message at a time per pod
---
// Horizontal Pod Autoscaler watches queue depth
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: order-processor-hpa
spec:
  scaleTargetRef:
    kind: Deployment
    name: order-processor
  minReplicas: 2
  maxReplicas: 20
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        averageUtilization: 70

The autoscaler doesn't need to understand your business logic. It just watches CPU or (better) a custom metric like queue depth, and adds workers when load increases. When the spike passes, it removes them. This is how platforms handle Black Friday spikes or viral events without manual intervention.

Handle Duplicates and Ordering Carefully

Here's where competing consumers gets tricky. Because messages can be redelivered (if a worker crashes), your code might process the same message twice. Your system must tolerate this. Ideally, operations are idempotent: running them twice produces the same result as running once.

In e-commerce, when a customer confirms checkout, we generate a PaymentRequested event. The event goes to a queue with a unique ID. If the worker crashes after charging the card but before updating inventory, the event redelivers, and we charge again—unless our code is idempotent. Here's the safe way:

async handlePaymentRequested(event: PaymentRequestedEvent) {
  // Check if we've already processed this event
  const existing = await this.database.payments.findOne({
    eventId: event.id,
  });
  
  if (existing) {
    console.log(`Payment ${event.id} already processed`);
    return; // Exit early, don't charge again
  }
  
  // Process the payment
  const charge = await this.stripe.charges.create({
    amount: event.amount,
    currency: 'usd',
    idempotencyKey: event.id, // Double protection
  });
  
  // Record it
  await this.database.payments.create({
    eventId: event.id,
    chargeId: charge.id,
    amount: event.amount,
    processedAt: new Date(),
  });
}

We store a record keyed by event ID. If the message redelivers, we check first and skip. We also use Stripe's idempotencyKey—Stripe itself won't charge twice for the same key. This layered defense is standard practice.

Ordering is the other gotcha. With competing consumers, you lose message ordering guarantees. Message A arrives before Message B, but consumer 1 might process B faster than consumer 2 processes A. If your logic depends on order (e.g., "deduct funds, then check balance"), you're in trouble.

The solution: for messages that must stay ordered, use a single queue per logical entity. In ride-hailing, we needed driver state updates in order—location, then rating, then availability change. We keyed messages by driver ID and routed them to a partitioned queue. Each driver's messages went to one partition, processed by one consumer, preserving order. Non-critical messages (logs, analytics) went to a global competing-consumer queue.

// Route based on driver ID
function getPartition(driverId: string, partitionCount: number): number {
  const hash = driverId.split('').reduce((h, c) => {
    return ((h << 5) - h) + c.charCodeAt(0);
  }, 0);
  return Math.abs(hash) % partitionCount;
}

// Producer
async publishDriverEvent(event: DriverEvent) {
  const partition = getPartition(event.driverId, 10);
  await this.queue.publish(`driver-events-partition-${partition}`, event);
}

// Consumer: one worker per partition ensures ordering
async consumePartition(partitionId: number) {
  const queue = `driver-events-partition-${partitionId}`;
  // Single worker per partition = ordered processing
}

This hybrid approach lets you scale globally while preserving order where it matters. The trade-off: more queue management complexity. Worth it when correctness is critical.

Common Mistakes

• Not setting prefetch: Workers grab too many messages at once, causing uneven load distribution. Always use prefetch(1) or similar.
• Acknowledging too early: If you ack before persisting results, a crash loses the work. Ack only after your database confirms the change.
• Assuming exactly-once without idempotency: Competing consumers give you at-least-once semantics. Build idempotent logic (deduplication by event ID, idempotency keys) to achieve exactly-once behavior.
• Ignoring ordering requirements: If your business logic depends on message order, competing consumers will break it. Use partitioned queues for entities needing order, competing consumers for independent work.
• Not monitoring queue depth: Deploy autoscaling based on queue depth, not just CPU. CPU can be misleading—fast workers on light messages won't spike CPU even if queue backs up.

Key Takeaways

• Competing consumers let multiple workers process the same queue independently, achieving horizontal scaling without complex coordination.
• Use prefetch limits (like prefetch(1)) to ensure even load distribution across workers.
• Acknowledge messages only after durable persistence; crashes before ack cause redelivery, which is expected and correct.
• Build idempotent operations and use event IDs or idempotency keys to tolerate duplicate processing.
• For messages requiring strict ordering, use partitioned queues (one worker per partition); reserve competing consumers for independent, order-agnostic work.
• Monitor queue depth and scale workers dynamically; autoscaling makes this nearly hands-off once configured.
• This pattern powers most production systems handling asynchronous, high-volume work—ride-hailing, e-commerce, trading platforms, streaming platforms.