Webhook handling best practices: lessons from production

Webhook handling best practices: lessons from production

We've all seen the payment that went missing, the refund that never fired, or the order status that got stuck in limbo—often because a webhook failed silently. Webhooks are how payments really complete: they're the asynchronous channel through which Stripe tells your system that a charge succeeded, a customer disputed a transaction, or a subscription renewed. In this article, we'll walk through webhook handling best practices: lessons from production, drawing on real scenarios from e-commerce and SaaS teams that process millions in transactions daily.

Respond fast — Return 200 quickly; process in a background job

The golden rule of webhook handling is simple: return an HTTP 200 status code as fast as possible. Stripe considers a webhook delivered only when it receives a 2xx response within a few seconds. If your handler blocks on business logic—calling external APIs, writing large reports, sending emails—Stripe will time out and retry the webhook repeatedly, creating duplicate work and cascading failures.

The pattern is straightforward: acknowledge receipt immediately, then defer the real work.

// Express + Node.js example
app.post('/webhooks/stripe', express.raw({type: 'application/json'}), async (req, res) => {
  const sig = req.headers['stripe-signature'];
  let event;

  try {
    event = stripe.webhooks.constructEvent(req.body, sig, process.env.STRIPE_WEBHOOK_SECRET);
  } catch (err) {
    return res.status(400).send(`Webhook Error: ${err.message}`);
  }

  // Queue the event for async processing
  await queue.add('process-stripe-event', event);

  // Return 200 immediately
  res.status(200).json({ received: true });
});

// Separate background job
queue.process('process-stripe-event', async (job) => {
  const event = job.data;

  switch (event.type) {
    case 'charge.succeeded':
      await handleChargeSucceeded(event.data.object);
      break;
    case 'invoice.payment_failed':
      await handleInvoicePaymentFailed(event.data.object);
      break;
  }
});

We typically use a job queue (Bull, RQ, Celery) or a reliable message broker (SQS, Kafka) for this. In production, we've seen teams lose transactions because they tried to update inventory, charge a commission, and notify a warehouse—all synchronously within the webhook handler. One slow third-party API call cascaded into Stripe retries, duplicate charges, and a weekend incident.

Idempotent processing — Store event ids and skip duplicates

Stripe will retry webhooks that fail to return a 200 response. Network hiccups, brief database outages, or load spikes can cause retries. If your webhook handler isn't idempotent, you'll process the same payment twice, create duplicate invoices, or double-charge customers. This is perhaps the most common and most damaging webhook mistake we see.

The fix is to store the Stripe event ID and check for duplicates before processing:

// Check if we've already processed this event
async function processWebhookEvent(event) {
  const existingRecord = await db.query(
    'SELECT id FROM stripe_events WHERE event_id = $1',
    [event.id]
  );

  if (existingRecord.length > 0) {
    console.log(`Event ${event.id} already processed, skipping.`);
    return;
  }

  // Process the event
  try {
    switch (event.type) {
      case 'charge.succeeded':
        await handleChargeSucceeded(event.data.object);
        break;
      // ... other cases
    }

    // Mark as processed
    await db.query(
      'INSERT INTO stripe_events (event_id, event_type, processed_at) VALUES ($1, $2, NOW())',
      [event.id, event.type]
    );
  } catch (err) {
    // Don't mark as processed; let it retry
    throw err;
  }
}

This simple pattern—store the event ID in a database table with a unique constraint—acts as a deduplication layer. The database insert acts as an atomic guard: either the event is processed and recorded, or neither happens. If the same event arrives again (via retry), the duplicate check catches it before any business logic runs.

We also recommend storing the full event payload. It makes debugging production issues much faster: you can replay or inspect any webhook without asking Stripe for logs.

Out-of-order events — Fetch the latest object state instead of trusting order

Here's a scenario that catches many teams off guard: you receive an invoice.payment_failed event, then an invoice.payment_succeeded event a second later. But due to a slow database, or a transient retry, those webhooks arrive out of order—success first, then failure. If you blindly apply the event state to your database, you'll end up with an invoice marked as failed when it actually succeeded.

The solution is to treat webhook events as signals to fetch fresh state, not as the source of truth:

async function handleInvoiceWebhook(event) {
  const invoiceId = event.data.object.id;

  // Fetch the current state from Stripe, don't trust the webhook payload
  const currentInvoice = await stripe.invoices.retrieve(invoiceId);

  // Now make decisions based on the authoritative state
  if (currentInvoice.paid) {
    await markInvoiceAsPaid(invoiceId);
  } else if (currentInvoice.status === 'draft') {
    await markInvoiceAsDraft(invoiceId);
  }

  // Only update fields we're certain about
  await db.query(
    `UPDATE invoices 
     SET stripe_status = $1, amount_total = $2, updated_at = NOW() 
     WHERE stripe_id = $3`,
    [currentInvoice.status, currentInvoice.amount_total, invoiceId]
  );
}

This pattern adds a small API call overhead per webhook (typically under 50ms), but it's worth it. You're querying Stripe's system of record for the ground truth, not relying on message ordering. In high-volume systems, this shields you from race conditions and ensures your data stays consistent even if webhooks are processed out of order or in parallel.

Retries — Stripe retries failed deliveries for days

Stripe doesn't give up easily. If your webhook handler returns a non-2xx status or times out, Stripe retries with exponential backoff over several days. This is a feature, not a bug—it means temporary infrastructure problems won't cause lost events. But it also means you need to be prepared for very old retries to suddenly arrive, and your idempotency guard must be robust.

Understanding Stripe's retry schedule helps you set realistic alert thresholds. If your monitoring dashboard fires an alert every time it sees a webhook retry, you'll get paged constantly. Instead, focus on the raw failure rate: if more than, say, 1% of webhooks fail within the first hour, that's a real problem. A single retry two days later is normal.

We recommend logging webhook deliveries and their outcomes:

async function processWebhookEvent(event) {
  const startTime = Date.now();
  let outcome = 'success';

  try {
    await deduplicateAndProcess(event);
  } catch (err) {
    outcome = 'failed';
    console.error(`Webhook processing failed for ${event.id}:`, err);
    throw err; // Let it retry
  } finally {
    const duration = Date.now() - startTime;
    await db.query(
      `INSERT INTO webhook_deliveries 
       (event_id, outcome, duration_ms, created_at) 
       VALUES ($1, $2, $3, NOW())`,
      [event.id, outcome, duration]
    );
  }
}

With this data, you can see patterns: Are failures clustered around a specific event type? Do they happen at particular times of day (maybe your database does backups)? Are retries succeeding? This operational visibility is invaluable for troubleshooting in production.

Common mistakes

• Blocking on synchronous logic: Validating credit card details, calling a fraud service, or fetching user records inside the webhook handler. Always queue the work.
• Trusting event order: Assuming that charge.succeeded always arrives before invoice.paid. Fetch fresh state from Stripe to make business decisions.
• Forgetting to verify the signature: Always validate that the webhook came from Stripe using the HMAC signature and your webhook secret. Stripe provides helpers for this in all major SDKs.
• Not handling missing events: If a webhook is lost (network partition, your service down), Stripe will retry, but there's a finite retry window. Build a reconciliation job that periodically queries Stripe's API to find any unprocessed invoices, charges, or subscriptions.
• Treating the webhook payload as immutable: The payload contains a snapshot of the object at the time the event was generated. Objects change; refunds happen, disputes are resolved. Always re-fetch to get the current state.

Key takeaways

• Return HTTP 200 immediately; defer business logic to a background job or queue to avoid timeout retries.
• Store the Stripe event ID and check for duplicates before processing; this prevents double-charges and duplicate side effects.
• Treat webhooks as signals, not as source of truth; fetch the latest object state from Stripe's API to handle out-of-order delivery and race conditions.
• Stripe retries over days with exponential backoff; use idempotency guards and logging to stay robust across retries and gracefully handle edge cases.
• Verify webhook signatures using HMAC and your webhook secret to ensure authenticity.
• Build a reconciliation job to catch any webhooks that were lost; use Stripe's API to periodically scan for unprocessed objects.