Handling payment webhooks at scale: a step-by-step guide

The Problem with Payment Webhooks at Scale

When you're processing thousands of payment events per minute, a naive webhook handler becomes your bottleneck. We've all seen it: a single slow database write blocks your entire queue, retry storms cascade through your system, and suddenly you're losing critical payment confirmations. Handling payment webhooks at scale requires you to think differently about queueing, persistence, and worker orchestration. This guide walks you through the production patterns we've used across e-commerce and SaaS platforms to stay reliable when payment volume grows.

Separate Queue Per Event Type If Needed

Not all payment events are created equal. A charge.succeeded event requires different logic than a charge.refunded event, and both have different urgency profiles than customer.subscription.updated. Running them through a single queue means a backlog of low-priority subscription updates can starve your critical charge handlers.

The solution is to route events to separate queues based on type, allowing you to size workers independently. Your charge handlers might need 10 workers running continuously, while subscription handlers could run 2 workers on a scaled schedule. This also makes monitoring simpler: you can alert on queue lag per event type rather than guessing where bottlenecks live.

Here's how we structure this in practice:

// Queue routing logic in your webhook handler
const QUEUE_CONFIG = {
  'charge.succeeded': { queue: 'payments-critical', priority: 1 },
  'charge.failed': { queue: 'payments-critical', priority: 2 },
  'charge.refunded': { queue: 'payments-critical', priority: 2 },
  'customer.subscription.updated': { queue: 'subscriptions', priority: 3 },
  'customer.subscription.deleted': { queue: 'subscriptions', priority: 3 },
  'invoice.payment_succeeded': { queue: 'invoicing', priority: 2 },
};

export async function handleWebhook(event: Stripe.Event) {
  const config = QUEUE_CONFIG[event.type];
  
  if (!config) {
    console.warn(`Unknown event type: ${event.type}`);
    return { status: 200 }; // Still ack to Stripe
  }

  await messageQueue.enqueue({
    queueName: config.queue,
    payload: event,
    priority: config.priority,
    maxRetries: 3,
  });

  return { status: 202, message: 'Event queued' };
}

We return 202 immediately after enqueueing, never waiting for handler completion. This keeps webhook acknowledgment fast and decouples event receipt from processing. Stripe expects a 2xx response within 5 seconds; anything slower risks retries you don't want.

Batch Database Writes

The biggest performance killer we see is writing to the database once per event. When you're handling 2,000 charge events per minute, that's 2,000 individual insert statements. Even with connection pooling, the overhead is brutal: transaction overhead, index updates, lock contention all multiply.

Batching transforms this. Instead of writing immediately, accumulate events in memory for 100ms or until you hit a batch size threshold (say, 500 events), then flush them all at once. A single batch insert of 500 rows is orders of magnitude faster than 500 individual inserts.

Here's a batching worker implementation:

class PaymentEventBatcher {
  private batch: ProcessedEvent[] = [];
  private readonly batchSize = 500;
  private readonly flushIntervalMs = 100;
  private flushTimer: NodeJS.Timeout | null = null;

  constructor(private db: Database) {}

  async addEvent(event: Stripe.Event) {
    const processed = await this.processEvent(event);
    this.batch.push(processed);

    if (this.batch.length >= this.batchSize) {
      await this.flush();
    } else if (!this.flushTimer) {
      this.flushTimer = setTimeout(() => this.flush(), this.flushIntervalMs);
    }
  }

  private async flush() {
    if (this.flushTimer) {
      clearTimeout(this.flushTimer);
      this.flushTimer = null;
    }

    if (this.batch.length === 0) return;

    const events = this.batch.splice(0);
    try {
      await this.db.query(
        `INSERT INTO payment_events (event_id, customer_id, amount, type, status, created_at)
         VALUES ${events.map((_, i) => `($${i * 5 + 1}, $${i * 5 + 2}, $${i * 5 + 3}, $${i * 5 + 4}, $${i * 5 + 5})`).join(', ')}
         ON CONFLICT (event_id) DO NOTHING`,
        events.flatMap(e => [e.id, e.customerId, e.amount, e.type, 'processed'])
      );
    } catch (error) {
      console.error('Batch insert failed, requeuing:', error);
      // Push events back to queue for retry
      for (const event of events) {
        await messageQueue.requeue(event);
      }
    }
  }
}

The ON CONFLICT DO NOTHING clause handles duplicate events safely. Stripe may deliver the same webhook multiple times, especially during retries, so idempotency is non-negotiable. We store the Stripe event ID and treat it as the unique constraint.

Beyond inserts, batching works for updates too. If you're updating order statuses based on charge events, accumulate those updates and apply them in one transaction. The key principle: minimize database round trips, not individual operations.

Monitor Queue Lag

You can't optimize what you don't measure. Queue lag—the time between when an event enters the queue and when a worker processes it—is your canary in the coal mine. When lag grows, you're losing ground. When it's steady at sub-second, you're healthy.

We instrument lag at multiple levels:

• Queue depth per type: How many events are waiting? Growing depth with constant lag means workers can't keep up.
• Worker throughput: Events processed per second per worker type. Helps you know if you need more workers.
• P95/P99 latency: Not just averages. One slow event out of a hundred means customer-facing timeouts.
• Dead letter rate: How many events are failing after retries? This is a critical alarm.

Here's instrumentation we'd wire up:

class QueueMonitor {
  constructor(private metrics: MetricsClient) {}

  async monitorQueue(queueName: string, intervalSec: number = 10) {
    setInterval(async () => {
      const depth = await messageQueue.getQueueDepth(queueName);
      const lag = await messageQueue.getAverageLag(queueName);
      const throughput = await messageQueue.getThroughput(queueName);

      this.metrics.gauge(`queue.depth`, depth, { queue: queueName });
      this.metrics.gauge(`queue.lag_ms`, lag, { queue: queueName });
      this.metrics.gauge(`queue.throughput_per_sec`, throughput, { queue: queueName });

      if (depth > 5000) {
        this.metrics.alert('High queue depth detected', { queue: queueName });
      }
      if (lag > 5000) {
        this.metrics.alert('Queue lag exceeding threshold', { queue: queueName });
      }
    }, intervalSec * 1000);
  }
}

We set up dashboards that show queue depth and lag side by side. If lag is climbing but depth is flat, something is slow in the handler itself. If both are climbing, you need more workers. This distinction is critical for knowing where to optimize.

Scale Workers Horizontally

Vertical scaling—bigger machines—helps only until it doesn't. A single process consuming from a queue can only do so much. True scaling means running more worker processes, ideally across multiple machines.

In production, we typically run workers in containers (Docker) orchestrated by Kubernetes or ECS. Each worker process is stateless: it pulls an event from the queue, processes it, acknowledges it back. If a worker crashes, another one picks up the work. This makes scaling trivial—just increase the replica count.

A typical horizontal scaling strategy looks like this:

// Kubernetes deployment example (conceptual)
// Adjust replicas based on queue.lag_ms metric
const workerConfig = {
  'payments-critical': {
    minReplicas: 3,
    maxReplicas: 20,
    targetLagMs: 1000,
  },
  'subscriptions': {
    minReplicas: 1,
    maxReplicas: 5,
    targetLagMs: 5000,
  },
  'invoicing': {
    minReplicas: 1,
    maxReplicas: 8,
    targetLagMs: 3000,
  },
};

// Auto-scaler pseudocode
async function autoScale(queueName: string) {
  const config = workerConfig[queueName];
  const currentLag = await metrics.getQueueLag(queueName);
  const currentReplicas = await k8s.getReplicaCount(`worker-${queueName}`);

  if (currentLag > config.targetLagMs && currentReplicas < config.maxReplicas) {
    await k8s.scale(`worker-${queueName}`, currentReplicas + 2);
  } else if (currentLag < config.targetLagMs * 0.3 && currentReplicas > config.minReplicas) {
    await k8s.scale(`worker-${queueName}`, currentReplicas - 1);
  }
}

The key insight: don't scale uniformly. Critical payment handlers need aggressive minimum replicas and fast scaling. Notification queues can scale more conservatively. This keeps costs down while protecting the critical path.

We also run workers with proper resource requests and limits to prevent one runaway worker from starving others. A memory limit ensures a handler that leaks memory gets killed and restarted before it destabilizes the node.

Common Mistakes

• Synchronous processing in webhooks: Writing to the database directly in your webhook handler. This blocks the HTTP thread and risks timeout. Always queue immediately.
• Ignoring idempotency: Processing the same Stripe event twice because you didn't check if it already existed. Always use the Stripe event ID as a unique constraint.
• Single queue for all events: Mixing critical payments with low-priority updates means delays in one impact the other. Separate queues give you control.
• Not monitoring lag: Flying blind. You won't know you have a problem until customers complain. Instrument queue depth and lag from day one.
• Hard-coded worker counts: Scaling manually via config is painful. Implement auto-scaling based on queue metrics so you can sleep at night.
• Batching without timeouts: If you batch but never flush (e.g., you crash before reaching batch size), events get lost. Always flush on a timer, not just on size.

Key Takeaways

• Route payment events to separate queues by type so you can scale critical handlers independently from low-priority ones.
• Batch database writes to reduce overhead; flush on both size (e.g., 500 events) and time (e.g., 100ms) thresholds.
• Monitor queue lag, queue depth, and throughput per queue type; these metrics tell you exactly where to optimize.
• Scale workers horizontally across multiple processes and machines; use Kubernetes or similar for stateless auto-scaling.
• Ensure idempotency by storing Stripe event IDs and using unique constraints; retries are inevitable.
• Keep webhook handlers fast: queue immediately, return 2xx within 5 seconds, let workers do the real work asynchronously.