A webhook dispatcher: a practical Design Patterns guide

Webhook reliability is harder than it looks

Most teams underestimate the complexity of sending webhooks at scale. You're not just making HTTP calls—you're building a fault-tolerant delivery system that must handle network failures, rate limits, endpoint outages, and downstream retry logic. In this guide, we'll walk through a webhook dispatcher: a practical Design Patterns guide that covers queuing, security, resilience, and lifecycle management. By the end, you'll have a production-ready pattern that keeps customers' integrations healthy and your infrastructure stable.

Queue each delivery

The first mistake teams make is dispatching webhooks synchronously during request handling. When your customer's endpoint is slow or offline, you block your entire request pipeline. Instead, treat each webhook delivery as a discrete job that gets queued immediately.

Use a persistent queue—Redis with a delayed queue library, RabbitMQ, AWS SQS, or even your database with a polling worker. The queue decouples your core business logic from delivery concerns. When an event fires (e.g., an order completes), you enqueue a delivery job and return success to the caller within milliseconds.

// Event handler in your core service
async function handleOrderCompleted(orderId: string): Promise<void> {
  const order = await orders.findById(orderId);
  
  // Update internal state synchronously
  await orders.markComplete(orderId);
  
  // Enqueue webhook delivery asynchronously
  await webhookQueue.enqueue({
    type: 'order.completed',
    timestamp: Date.now(),
    payload: {
      orderId: order.id,
      amount: order.total,
      customerId: order.customerId,
    },
  });
}

// Separate worker processes queue items
webhookQueue.on('message', async (job) => {
  const webhook = await webhooks.findByType(job.type);
  if (!webhook) return;
  
  await dispatcher.send(webhook.url, job.payload);
});

By queuing, you also gain visibility. You can inspect pending deliveries, replay failed ones, and add metrics per endpoint. Teams often discover integration issues only when customers complain—a queue gives you the truth.

Sign payloads with HMAC

Customers need to verify that webhooks actually came from you, not from an attacker spoofing your IP or DNS. Use HMAC (Hash-based Message Authentication Code) to cryptographically sign each payload.

Store a unique secret for each customer's webhook endpoint. When dispatching, compute an HMAC digest of the request body using that secret and your signing key. Include the digest in a header (commonly X-Signature or X-Hub-Signature). The customer recomputes the digest on their end; if it matches, the payload is authentic.

import crypto from 'crypto';

async function sendWithSignature(
  url: string,
  payload: object,
  secret: string
): Promise<Response> {
  const body = JSON.stringify(payload);
  
  // Compute HMAC-SHA256 signature
  const signature = crypto
    .createHmac('sha256', secret)
    .update(body)
    .digest('hex');
  
  const response = await fetch(url, {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      'X-Signature': `sha256=${signature}`,
      'X-Timestamp': Date.now().toString(),
    },
    body,
  });
  
  return response;
}

// Customer's verification on their end
function verifyWebhook(request: Request, secret: string): boolean {
  const signature = request.headers.get('X-Signature');
  const body = request.body;
  
  const computed = crypto
    .createHmac('sha256', secret)
    .update(body)
    .digest('hex');
  
  return crypto.timingSafeEqual(
    Buffer.from(signature),
    Buffer.from(`sha256=${computed}`)
  );
}

Include a timestamp header too. Customers can reject payloads older than a threshold (e.g., 5 minutes) to guard against replay attacks. Never log or store full payloads; they may contain sensitive data. Log only the URL, timestamp, and response code.

Retry with backoff

Network hiccups are inevitable. An endpoint might timeout, return 500, or be temporarily offline. Naive retry logic (immediate retry in a loop) overloads the customer's server and burns resources. Instead, implement exponential backoff with jitter.

On first failure, retry after 2 seconds. On second failure, wait 4 seconds. Then 8, 16, 32, 60, 120 seconds. Add random jitter (±20%) to prevent thundering herd scenarios where thousands of workers retry at the same moment. Most platforms benefit from 3–5 retry attempts over 5–10 minutes.

interface WebhookDeliveryJob {
  id: string;
  url: string;
  payload: object;
  secret: string;
  attempt: number;
  maxAttempts: number;
  nextRetryAt?: number;
}

async function processDeliveryJob(job: WebhookDeliveryJob): Promise<void> {
  try {
    const response = await sendWithSignature(job.url, job.payload, job.secret);
    
    if (response.ok) {
      await markDeliverySuccess(job.id);
      return;
    }
    
    // 4xx errors are not retryable
    if (response.status < 500) {
      await markDeliveryFailed(job.id, response.status);
      return;
    }
    
    // 5xx errors warrant retry
    throw new Error(`Server error: ${response.status}`);
  } catch (error) {
    if (job.attempt < job.maxAttempts) {
      const backoffSeconds = Math.pow(2, job.attempt);
      const jitter = backoffSeconds * (0.8 + Math.random() * 0.4);
      
      await requeue(job, {
        ...job,
        attempt: job.attempt + 1,
        nextRetryAt: Date.now() + jitter * 1000,
      });
    } else {
      await markDeliveryFailed(job.id, `Max retries exceeded: ${error.message}`);
    }
  }
}

Track retry metadata in your database or queue. Log each attempt with the response status, error message, and scheduled retry time. This history is invaluable for debugging customer issues and understanding which endpoints are chronically failing.

Disable endpoints after repeated failures

If a customer's endpoint fails consistently—say, 10 deliveries in a row over an hour—continuing to hammer it is wasteful and looks aggressive. Instead, automatically disable the endpoint and notify the customer.

A disabled endpoint stops receiving webhooks until the customer explicitly re-enables it (ideally after fixing their server). Some teams use a soft disable: if failures exceed a threshold in a time window, pause deliveries for 24 hours, then retry once. If that succeeds, re-enable; if it fails again, mark it disabled and require manual intervention.

interface WebhookEndpoint {
  id: string;
  url: string;
  enabled: boolean;
  failureCount: number;
  lastFailureAt?: number;
  disabledAt?: number;
}

async function recordDeliveryFailure(
  endpointId: string,
  error: string
): Promise<void> {
  const endpoint = await webhooks.findById(endpointId);
  
  const updatedCount = endpoint.failureCount + 1;
  const now = Date.now();
  
  // Clear old failures outside a 1-hour window
  const oneHourAgo = now - 3600000;
  const recentFailures = endpoint.failureCount;
  // (In production, query actual failure timestamps from logs)
  
  if (recentFailures >= 5) {
    // Disable and notify
    await webhooks.update(endpointId, {
      enabled: false,
      disabledAt: now,
      failureCount: updatedCount,
    });
    
    await alerting.notifyCustomer(endpoint.customerId, {
      title: 'Webhook endpoint disabled',
      message: `Your webhook endpoint ${endpoint.url} has been disabled due to repeated failures.`,
      action: 'https://dashboard.example.com/webhooks/' + endpointId,
    });
  } else {
    await webhooks.update(endpointId, {
      failureCount: updatedCount,
      lastFailureAt: now,
    });
  }
}

Maintain a dashboard where customers can see their endpoint health: success rate, average response time, last 10 delivery attempts, and disable/enable controls. This transparency builds trust and speeds up issue resolution.

Common mistakes

• Synchronous dispatch: Sending webhooks inside request handlers blocks your app. Always queue first.
• No signature verification: Customers can't trust unsigned payloads. HMAC is table stakes for security.
• Naive retry logic: Retrying immediately or without backoff creates cascading failures and upset customers. Use exponential backoff with jitter.
• No endpoint lifecycle: Continuing to send to dead endpoints wastes resources. Disable after threshold failures and notify the customer.
• Insufficient observability: Log delivery attempts, response codes, and retry schedules. Without these, debugging customer issues is a guessing game.
• Testing gaps: Test timeout scenarios, 5xx responses, network partitions, and rate limits. Mock your queue and HTTP client to simulate failures reliably.

Key takeaways

• Queue every webhook delivery job instead of dispatching synchronously. This decouples your core logic from delivery concerns and gives you a retry mechanism.
• Sign payloads with HMAC-SHA256 and include a timestamp header. Customers verify signatures on receipt; unsigned webhooks cannot be trusted.
• Implement exponential backoff (2s, 4s, 8s, 16s, …) with jitter. Avoid thundering herd retries by randomizing retry times across your retry attempts.
• Automatically disable endpoints after repeated failures (e.g., 5 failures in 1 hour) and alert the customer. Re-enable only after manual verification or a successful test delivery.
• Build observability into your dispatcher: log every attempt, track endpoint health metrics, and expose a customer-facing dashboard. You'll catch issues faster and earn customer confidence.