Event replay and recovery: practical tips and trade-offs

Event replay and recovery: practical tips and trade-offs

Webhook failures are inevitable. Network timeouts, service crashes, and temporary outages mean your payment system will miss events from Stripe, leaving your orders incomplete or customer records stale. The difference between a resilient payment platform and a fragile one often comes down to how well you handle event replay and recovery—the ability to detect missed webhooks, retrieve them, and safely reprocess them without duplicating charges or refunds.

In this post, we'll walk through practical patterns we've used in production across e-commerce and SaaS platforms: listing events from the Stripe API, reprocessing with idempotent handlers, storing raw events for internal replay, and alerting when processing lags. Each approach has trade-offs, and we'll cover the ones that matter most.

List events from the Stripe API

The simplest recovery path starts with the Stripe Events API. Every action on your account—charges, refunds, disputes, subscriptions—generates an event that Stripe stores for 30 days. If your webhook endpoint was down or flaky, you can query this API to find what you missed.

Here's a common scenario: your payment service crashes for 10 minutes during a deployment. In that window, Stripe sends webhook payloads to your endpoint, gets timeouts, and stops retrying. Your database still lacks those payment confirmations. The fix is to list events within that time window and reprocess them.

import Stripe from 'stripe';

const stripe = new Stripe(process.env.STRIPE_SECRET_KEY);

async function recoverMissedEvents(startTime: Date, endTime: Date) {
  const events = [];
  
  for await (const event of stripe.events.list({
    created: {
      gte: Math.floor(startTime.getTime() / 1000),
      lte: Math.floor(endTime.getTime() / 1000),
    },
    // Filter by event type if you know which ones you care about
    // type: 'charge.succeeded',
    limit: 100,
  })) {
    events.push(event);
  }
  
  return events;
}

// Usage after an incident
const missedEvents = await recoverMissedEvents(
  new Date('2024-01-15T14:30:00Z'),
  new Date('2024-01-15T14:40:00Z')
);

console.log(`Found ${missedEvents.length} events to replay`);
for (const event of missedEvents) {
  console.log(`${event.type}: ${event.id}`);
}

The main advantage here is simplicity: no extra infrastructure or database schema changes needed. The downside is latency. Every time you want to recover, you're making multiple API calls to Stripe, which adds seconds or minutes to your recovery process. For non-critical events, that's fine. For payment confirmations, you want faster.

Also note the 30-day window. Stripe doesn't keep events forever. If your outage is detected weeks later, you've lost the trail.

Reprocess with idempotent handlers

Once you have events to replay, the second challenge is safety. You cannot simply re-run your webhook handler on every event—you risk creating duplicate charges, sending duplicate emails, or double-crediting accounts.

The solution is idempotent event processing: your handlers must produce the same result whether called once or multiple times with the same input. This typically means checking if an action already completed before executing it again.

Consider a charge.succeeded event. Your handler updates the order status and sends a confirmation email. If you reprocess it, you need to check: has this charge already been marked as succeeded? If yes, skip the duplicate email but confirm the order is in the right state.

async function handleChargeSucceeded(event: Stripe.Event) {
  const charge = event.data.object as Stripe.Charge;
  
  // Idempotency key: unique identifier for this event
  const idempotencyKey = event.id;
  
  // Check if we've already processed this event
  const existingRecord = await db.eventLog.findUnique({
    where: { stripeEventId: idempotencyKey },
  });
  
  if (existingRecord) {
    console.log(`Event ${idempotencyKey} already processed. Skipping.`);
    return { processed: false, reason: 'duplicate' };
  }
  
  try {
    // Get the order
    const order = await db.order.findUnique({
      where: { stripeChargeId: charge.id },
    });
    
    if (!order) {
      console.error(`Order not found for charge ${charge.id}`);
      return { processed: false, reason: 'order_not_found' };
    }
    
    // Update order status
    const updatedOrder = await db.order.update({
      where: { id: order.id },
      data: {
        status: 'confirmed',
        paidAt: new Date(charge.created * 1000),
      },
    });
    
    // Send confirmation email (idempotent on our side too)
    await emailService.sendConfirmation(order.customerId, order.id);
    
    // Log this event as processed
    await db.eventLog.create({
      data: {
        stripeEventId: idempotencyKey,
        eventType: event.type,
        processedAt: new Date(),
        orderId: order.id,
      },
    });
    
    return { processed: true, orderId: order.id };
  } catch (error) {
    console.error(`Failed to process event ${idempotencyKey}:`, error);
    throw error;
  }
}

The eventLog table is your safety net. By storing a record of each processed Stripe event ID, you can detect replays and exit early. This must be written atomically with your business logic, ideally in the same database transaction, to avoid race conditions.

A key trade-off: you're adding database overhead. Every event now requires at least one extra query to check if it's been seen before. In high-volume scenarios (thousands of events per second), this can become a bottleneck. Use indexing on stripeEventId and consider caching frequently checked events in Redis.

Store raw events to replay internally

Relying solely on Stripe's API for recovery is risky. Stripe's 30-day window is firm, and listing events during an outage adds latency. A more resilient approach is to store every webhook payload you receive in your own database or event log, then replay from there.

This gives you a few advantages: unlimited replay window (you control the retention), faster replay (no external API calls), and a local audit trail of everything Stripe sent you. The trade-off is storage and the complexity of managing another table.

// Webhook handler that stores the raw event
async function handleStripeWebhook(req: express.Request, res: express.Response) {
  const sig = req.headers['stripe-signature'] as string;
  let event: Stripe.Event;
  
  try {
    event = stripe.webhooks.constructEvent(
      req.body,
      sig,
      process.env.STRIPE_WEBHOOK_SECRET
    );
  } catch (err) {
    return res.status(400).send(`Webhook Error: ${err.message}`);
  }
  
  try {
    // Store the raw event immediately, before processing
    await db.stripeEventStore.create({
      data: {
        eventId: event.id,
        eventType: event.type,
        rawPayload: JSON.stringify(event),
        receivedAt: new Date(),
        processedAt: null, // Will update after successful processing
      },
    });
    
    // Then process it
    const result = await handleChargeSucceeded(event);
    
    // Mark as processed
    await db.stripeEventStore.update({
      where: { eventId: event.id },
      data: { processedAt: new Date() },
    });
    
    res.json({ received: true });
  } catch (error) {
    // If processing failed, leave processedAt as null
    // We'll retry this event later
    console.error(`Webhook processing failed for event ${event.id}:`, error);
    res.status(500).json({ error: 'Processing failed' });
  }
}

// Later, replay unprocessed events
async function replayUnprocessedEvents() {
  const unprocessed = await db.stripeEventStore.findMany({
    where: { processedAt: null },
    orderBy: { receivedAt: 'asc' },
    take: 100,
  });
  
  for (const record of unprocessed) {
    try {
      const event: Stripe.Event = JSON.parse(record.rawPayload);
      await handleChargeSucceeded(event);
      
      await db.stripeEventStore.update({
        where: { eventId: record.eventId },
        data: { processedAt: new Date() },
      });
      
      console.log(`Replayed event ${record.eventId}`);
    } catch (error) {
      console.error(`Failed to replay event ${record.eventId}:`, error);
      // Don't throw; continue with next event
    }
  }
}

This approach decouples your webhook reception from processing. You store first, process later. If processing fails, you have a queue of events ready to retry. This is especially valuable for async scenarios where you want to process events in bulk or with custom logic.

Storage is the main consideration. A typical Stripe event JSON is 1–3 KB. At 1,000 events per day, you're looking at 1–3 MB per day, or 30–90 MB per month. That's manageable for most databases, but with cleanup policies you can prune events older than 90 days to save space.

Alert when processing lags

Storing events is only useful if you know when things go wrong. A production system should monitor event processing latency and alert on delays. If your queue of unprocessed events grows or events sit unprocessed for longer than expected, you need to know immediately.

// Health check: monitor event processing lag
async function checkEventProcessingHealth() {
  const oldestUnprocessed = await db.stripeEventStore.findFirst({
    where: { processedAt: null },
    orderBy: { receivedAt: 'asc' },
  });
  
  if (!oldestUnprocessed) {
    // All events processed, we're good
    return { healthy: true, lag: 0 };
  }
  
  const lagMs = Date.now() - oldestUnprocessed.receivedAt.getTime();
  const lagSeconds = Math.round(lagMs / 1000);
  
  // Alert if any event is unprocessed for more than 5 minutes
  const ALERT_THRESHOLD_MS = 5 * 60 * 1000;
  
  if (lagMs > ALERT_THRESHOLD_MS) {
    await alerting.sendAlert({
      severity: 'critical',
      title: 'Stripe event processing lag detected',
      message: `Oldest unprocessed event is ${lagSeconds}s old. ` +
               `Event ID: ${oldestUnprocessed.eventId}`,
      tags: ['stripe', 'payments'],
    });
    
    return { healthy: false, lag: lagSeconds };
  }
  
  return { healthy: true, lag: lagSeconds };
}

// Run this check periodically (every 1 minute)
setInterval(checkEventProcessingHealth, 60 * 1000);

// Also count unprocessed events
async function countUnprocessedEvents() {
  const count = await db.stripeEventStore.count({
    where: { processedAt: null },
  });
  
  const BACKLOG_THRESHOLD = 100;
  if (count > BACKLOG_THRESHOLD) {
    await alerting.sendAlert({
      severity: 'warning',
      title: 'Stripe event processing backlog',
      message: `${count} events waiting to be processed`,
    });
  }
  
  return count;
}

Couple this monitoring with a dashboard that shows event lag and queue depth in real time. When alerts fire, your on-call engineer can quickly assess whether it's a temporary blip or a real outage.

The threshold depends on your use case. For subscriptions or critical charges, 5 minutes is too long. For transactional emails triggered by events, an hour might be acceptable. Tune based on your SLOs.

Common mistakes

• Forgetting that Stripe retries webhooks. Stripe will retry failed webhooks exponentially over 3 days. Your alert logic shouldn't assume immediate manual replay is needed; give Stripe time first. Only manually replay if you detect genuinely missed events after 24 hours.
• Not making email sends idempotent. You store events safely and check for duplicates in your database, but then your email service sends the same email twice because you don't check there. Either use an email service that supports idempotency keys, or track sent emails in your database.
• Storing the entire Stripe object without versioning. Stripe occasionally adds fields to event objects. If you serialize the full object and later code expects new fields, your replay might fail. Consider storing just the event ID and fetching the full object from Stripe at replay time, or carefully version your event schema.
• Assuming all events should be replayed the same way. Some events (like disputes) might need different replay logic than charges. Build your replay system to handle event-type-specific logic, not a one-size-fits-all handler.
• Not testing replay in staging. The only way to validate your idempotent logic is to actually replay events. Set up a staging environment with a copy of production data, trigger some Stripe test events, then manually replay them and verify nothing duplicates.

Key takeaways

• Start with Stripe's Events API for one-off recovery, but recognize its limitations: 30-day window and latency.
• Store raw webhook payloads in your database immediately upon receipt. This gives you an unlimited, fast replay source.
• Build idempotent event handlers using an event log table keyed on Stripe event ID. Check this table before processing to avoid duplicates.
• Make your downstream actions (emails, API calls, database updates) idempotent too. Idempotency at the handler level is not enough.
• Monitor event processing lag and backlog actively. Alert when events sit unprocessed beyond your SLO.
• Test replay scenarios in staging before they're needed in production. A well-rehearsed recovery procedure is worth far more than a perfect-in-theory one.
• Balance storage and retention costs. You don't need to keep events forever; 90 days of history covers most incident investigations.