Webhooks that don't lose data: best practices explained

Why webhooks fail and how to build resilient ones

Your payment processor just sent a webhook confirming a $50,000 transaction. Your endpoint crashed. The message got lost. Your revenue is now unreconciled. Webhooks that don't lose data: best practices explained sounds like a sidequest, but it's actually the core of running reliable integrations with third-party services like payment gateways, Shopify, Meta Lead Ads, and countless SaaS platforms.

In this guide, we'll walk through the patterns that Meta, Shopify, and payment processors themselves use to ensure critical data reaches your system intact, and how to implement them in your own infrastructure.

Respond 200 quickly; process asynchronously

The most common mistake teams make is treating the webhook endpoint as a processing endpoint. You read the payload, validate it, hit your database, call downstream services, and only then return a 200 status code. If any of that takes more than a few seconds, the webhook provider times out, assumes delivery failed, and retries. Most providers retry aggressively: after 1 second, 5 seconds, then exponential backoff up to hours later.

Shopify's webhook documentation is explicit about this: return 200 within 5 seconds. If you don't, they'll retry. Meta's Lead Ads webhooks have a 30-second timeout. Payment gateways like Stripe typically allow 10–30 seconds but expect you to respect that boundary.

The pattern is simple: acknowledge receipt immediately, then queue the work.

import { Router } from 'express';
import Bull from 'bull';

const webhookRouter = Router();
const processingQueue = new Bull('webhook-processing');

webhookRouter.post('/webhooks/payment', async (req, res) => {
  // Return 200 instantly
  res.status(200).send({ received: true });
  
  // Queue the actual work
  await processingQueue.add(req.body, {
    attempts: 5,
    backoff: {
      type: 'exponential',
      delay: 2000
    }
  });
});

// Process asynchronously
processingQueue.process(async (job) => {
  const payload = job.data;
  
  // Now we have time to:
  // - validate the signature
  // - fetch related records
  // - call APIs
  // - write to database
  // - log everything
  
  console.log('Processing webhook:', payload.id);
  await updateTransactionStatus(payload);
});

This pattern decouples acknowledgment from processing. Your endpoint is fast and reliable. The queue provides retry logic with backoff, so transient database locks or API timeouts don't cause data loss. Dead-letter queues catch jobs that fail repeatedly, and you can investigate them manually.

Verify signatures — anyone can POST to your endpoint

Your webhook endpoint is a public URL. Anyone with a network scanner can find it. They can craft POST requests that look legitimate. Without signature verification, attackers can inject fake transaction confirmations, fraudulent refunds, or cancel orders that never happened.

Every major provider signs their webhooks. Shopify uses HMAC-SHA256. Stripe uses HMAC-SHA256. Meta uses a signature header. The process is:

1. Provider hashes the raw request body with a secret key you both share
2. Provider includes that hash in a header (usually X-Shopify-Hmac-SHA256, t=... for Stripe, or X-Hub-Signature)
3. You hash the same body with the same secret
4. You compare: if they match, the request came from them

The critical detail: hash the raw request body as bytes, before any JSON parsing or normalization. Different JSON encoders produce different output even for identical data structures.

import crypto from 'crypto';
import express from 'express';

const app = express();

// Get raw body as buffer before JSON parsing
app.post(
  '/webhooks/shopify',
  express.raw({ type: 'application/json' }),
  async (req, res) => {
    const body = req.body; // This is a Buffer
    const hmacHeader = req.get('X-Shopify-Hmac-SHA256');
    const secret = process.env.SHOPIFY_WEBHOOK_SECRET;

    // Verify signature
    const hash = crypto
      .createHmac('sha256', secret)
      .update(body, 'utf8')
      .digest('base64');

    if (hash !== hmacHeader) {
      console.error('Invalid signature');
      return res.status(401).send('Unauthorized');
    }

    // Now safe to parse JSON
    const payload = JSON.parse(body.toString());
    
    res.status(200).send({ received: true });
    await processingQueue.add(payload);
  }
);

Store the secret in your environment, never in code. Rotate it periodically. If you suspect a compromise, regenerate it in your provider's dashboard and update your environment before old webhooks stop working.

Store the raw payload before parsing it

You've verified the signature. You're about to parse JSON and update your database. Then a bug in your parsing logic crashes the process. The webhook is lost — the provider won't resend it for hours, and you have no record of what was sent.

Store the entire raw payload in a webhook log table immediately after verification. Include the headers, the timestamp, and the provider name. This is your source of truth if anything goes wrong.

webhookRouter.post('/webhooks/payment', 
  express.raw({ type: 'application/json' }),
  async (req, res) => {
    const rawBody = req.body;
    const signature = req.get('X-Stripe-Signature');
    
    // Verify...
    if (!verifyStripeSignature(rawBody, signature)) {
      return res.status(401).send('Unauthorized');
    }

    // IMMEDIATELY store raw payload
    const webhookRecord = await db.webhookLogs.create({
      provider: 'stripe',
      rawPayload: rawBody.toString(),
      headers: JSON.stringify(req.headers),
      receivedAt: new Date(),
      status: 'received',
      idempotencyKey: null
    });

    res.status(200).send({ received: true });

    // Queue processing, include webhook record ID
    await processingQueue.add({
      webhookLogId: webhookRecord.id,
      payload: JSON.parse(rawBody.toString())
    });
  }
);

Later, as you process the webhook in your queue worker, update that record's status. If processing succeeds, mark it as processed. If it fails with a recoverable error, leave it as pending for the queue to retry. If it fails permanently, mark it as failed and alert your team.

This log becomes invaluable for debugging. You can replay webhooks manually, audit what was received versus what was acted upon, and detect if a provider is resending old data after an outage.

Providers send duplicates; build for replays

Webhooks are not guaranteed to arrive exactly once. They're guaranteed to arrive at least once. A provider might send the same webhook twice due to network issues, timeouts, or internal retries. Your endpoint might receive it, process it successfully, return 200, but the provider doesn't record receipt and sends it again 5 seconds later.

If your webhook handler isn't idempotent, processing the same payment confirmation twice could charge the customer twice or double-count revenue.

The solution: every webhook should have a unique identifier (provided by the provider) that you track. Before processing, check if you've already handled this ID. If so, return 200 without re-processing.

const processingQueue = new Bull('webhook-processing');

processingQueue.process(async (job) => {
  const { webhookLogId, payload } = job.data;
  const idempotencyKey = payload.id; // Stripe, Shopify, etc. provide this

  // Check if we've already processed this webhook
  const existing = await db.webhookLogs.findOne({
    idempotencyKey,
    status: 'processed'
  });

  if (existing) {
    console.log(`Webhook ${idempotencyKey} already processed, skipping`);
    return;
  }

  try {
    // Process the webhook
    if (payload.type === 'payment.success') {
      await db.transactions.create({
        externalId: payload.id,
        amount: payload.amount,
        status: 'confirmed',
        processedAt: new Date()
      });
    }

    // Mark as processed atomically
    await db.webhookLogs.update(
      { id: webhookLogId },
      { 
        status: 'processed',
        idempotencyKey,
        processedAt: new Date()
      }
    );

  } catch (error) {
    console.error('Failed to process webhook:', error);
    throw error; // Let queue retry
  }
});

The idempotency check must happen atomically with the processing. Use a database transaction or a unique constraint on idempotencyKey to ensure that even if two requests process in parallel, only one succeeds in creating a transaction record.

This pattern also enables manual replay. If you deploy a bug fix and need to reprocess webhooks from an hour ago, extract them from your webhook log, re-queue them, and they'll be handled idempotently.

Common mistakes

• Processing synchronously in the webhook handler: Timeouts cascade into retries, and you lose data when deployments restart your server mid-webhook.
• Skipping signature verification: A single line of code prevents fraud and injected data.
• Parsing JSON before verifying the signature: Malformed JSON could crash your handler before you ever validate authenticity.
• Forgetting to store the raw payload: When something goes wrong, you'll have no audit trail and no way to replay.
• Assuming webhooks arrive exactly once: They don't. Build idempotency into every handler, or face duplicate transactions.
• Not setting up dead-letter queues: When a webhook can't be processed after retries, it silently disappears. Set up alerts on DLQ messages.
• Storing secrets in environment files checked into git: Use a secrets manager (AWS Secrets Manager, HashiCorp Vault, or your platform's native option).

Key takeaways

• Return 200 immediately from your webhook endpoint; queue the actual work for asynchronous processing.
• Verify every webhook signature before trusting the data, and hash the raw body before JSON parsing.
• Store the complete raw payload and headers in a webhook log table for auditing and replay.
• Treat webhooks as at-least-once delivery: build idempotency using a unique identifier provided by the provider.
• Use a queue with retry logic and dead-letter queue handling to catch and investigate failures.
• Regularly test webhook handling by replaying old payloads and simulating provider retries.