Retry topics and dead-letter queues in Kafka & WebSockets

When a Kafka consumer encounters an error processing an event, the instinct is often to retry immediately—but that approach can paralyze your entire partition. Retry topics and dead-letter queues are the backbone of robust failure handling in event-driven systems, and they solve a critical problem: how do you recover from transient failures without blocking every event behind a problematic message?

Understanding the retry flow

The retry flow pattern I use across production systems follows a simple but effective progression: Main topic → Retry topic (delay 1m) → Retry topic (delay 10m) → Dead-letter topic. This tiered approach gives transient failures multiple chances to resolve while preventing infinite loops.

Here's how it works in practice. A consumer processes an event from the main topic. If it encounters a retriable error—say, a database connection timeout or a downstream service temporarily unavailable—instead of retrying synchronously, it publishes the event to a retry topic with a timestamp or delay indicator. A separate consumer group polls the retry topic. After a fixed delay (typically 1 minute for the first retry), it attempts reprocessing. If that fails again, it moves to a longer-delay retry topic (10 minutes). If it fails a third time, it lands in the dead-letter queue (DLQ) for manual inspection.

This design keeps your main partition unblocked. While a problematic event bounces through retries, the partition consumer continues processing subsequent events. You avoid the classic scenario where one poison message halts processing for thousands of events queued behind it.

Don't block the partition

Partition blocking happens when a consumer synchronously retries a failed event in place. You send a request, it fails, you wait 5 seconds, you retry, it fails again—meanwhile, the offset hasn't advanced, and every event behind this one is stuck.

I learned this lesson early in a ride-hailing system. A third-party geolocation service went down for 15 minutes. Our consumer attempted exponential backoff, retrying each failed location lookup for up to 2 minutes. During those 15 minutes, we couldn't process any new ride requests. Drivers were invisible, riders couldn't book. The solution: extract the problematic event to a separate queue, immediately commit the offset, and let the main consumer move on.

The technical win is simple but profound. By publishing to a retry topic instead of blocking, you maintain throughput on your main partition. Your consumer advances the offset, frees up memory, and processes the next batch. A dedicated retry consumer handles the backoff asynchronously, decoupled from the critical path.

Consider this flow in code:

async function handleEvent(event: Event): Promise<void> {
  try {
    await processEvent(event);
    // Success, commit offset
  } catch (error) {
    if (isRetriable(error)) {
      // Publish to retry topic, not local retry loop
      await producer.send({
        topic: 'events-retry-1m',
        messages: [{
          key: event.id,
          value: JSON.stringify({
            ...event,
            retryCount: (event.retryCount || 0) + 1,
            originalError: error.message,
            retryTimestamp: Date.now()
          })
        }]
      });
      // Commit offset and move on
      return;
    } else {
      // Non-retriable error, send to DLQ
      await producer.send({
        topic: 'events-dlq',
        messages: [{
          key: event.id,
          value: JSON.stringify({
            event,
            error: error.message,
            stack: error.stack,
            failedAt: new Date().toISOString()
          })
        }]
      });
      return;
    }
  }
}

The consumer processes the event, makes a decision, and moves forward—never blocking.

Classify errors

Not all errors deserve retries. A 400 Bad Request from an API won't fix itself by waiting 10 minutes. A network timeout might. The difference between transient and permanent errors is crucial to avoid wasting retries on doomed messages.

Transient errors are typically temporary and environmental: database connection timeouts, circuit breaker rejections, 5xx server errors, rate-limit responses that indicate temporary saturation. Permanent errors are logical failures: validation errors (malformed event structure), authentication failures, resource-not-found (404), and permission denials (403).

I classify errors at the point of failure with explicit decision logic:

function isRetriable(error: Error): boolean {
  // Network and availability issues
  if (error.code === 'ECONNREFUSED' || error.code === 'ETIMEDOUT') {
    return true;
  }
  
  // HTTP status codes
  if (error.statusCode) {
    // 5xx errors: server issue, likely transient
    if (error.statusCode >= 500) return true;
    
    // 429: rate limited, transient
    if (error.statusCode === 429) return true;
    
    // 4xx errors: client issue, permanent
    if (error.statusCode >= 400) return false;
  }
  
  // Circuit breaker open
  if (error.message.includes('circuit breaker open')) {
    return true;
  }
  
  // Database connection issues
  if (error.message.includes('connection refused') || 
      error.message.includes('ECONNREFUSED')) {
    return true;
  }
  
  // Default to non-retriable for safety
  return false;
}

This classification prevents retry storms. A validation error doesn't retry—it goes straight to the DLQ where a human can investigate. A timeout retries because the downstream service might recover.

Make DLQs actionable

A dead-letter queue that no one monitors is just a graveyard. In production, the DLQ must be observable, and recovery must be automated where safe.

I structure DLQ messages with full context: the original event, the error message, the stack trace, and metadata about when and where it failed. Then I build a replay tool—either a CLI command or dashboard—that lets operators reprocess specific messages after the underlying issue is fixed.

Here's what a well-structured DLQ consumer looks like:

async function handleDLQEvent(dlqMessage: DLQMessage): Promise<void> {
  const { event, error, failedAt, retryCount } = dlqMessage;
  
  // Log to structured logging with full context
  logger.error('Event in DLQ', {
    eventId: event.id,
    eventType: event.type,
    error: error,
    failedAt,
    retryCount,
    timestamp: new Date().toISOString()
  });
  
  // Store in DLQ database for later replay
  await dlqStore.insert({
    eventId: event.id,
    originalEvent: event,
    errorMessage: error,
    failedAttempts: retryCount,
    status: 'pending', // pending, resolved, replayed
    createdAt: new Date()
  });
  
  // Alert operations if retry count exceeded threshold
  if (retryCount >= 3) {
    await alerting.send({
      channel: 'incidents',
      message: `Event ${event.id} exhausted retries`,
      severity: 'warning'
    });
  }
}

Beyond storage, provide a replay mechanism. When the underlying issue is resolved (database recovered, API fixed, service scaled), operators should be able to reprocess those messages in bulk:

async function replayDLQEvent(eventId: string): Promise<void> {
  const dlqRecord = await dlqStore.findById(eventId);
  
  // Republish to main topic for reprocessing
  await producer.send({
    topic: 'main-events-topic',
    messages: [{
      key: dlqRecord.eventId,
      value: JSON.stringify(dlqRecord.originalEvent)
    }]
  });
  
  // Update DLQ record status
  await dlqStore.update(eventId, { status: 'replayed' });
}

This transforms the DLQ from a black hole into a safety net—events are captured, operators are notified, and recovery is self-service.

Common mistakes

• Infinite retries on retriable errors: If you don't cap retry attempts, a flaky service can generate thousands of duplicate messages. Always set a maximum retry count (I use 3) before routing to the DLQ.
• Retrying too aggressively: A 30-second retry backoff is too fast for most issues. Aim for 1 minute minimum for the first retry, 10 minutes for the second. This gives transient issues time to resolve without overwhelming the system.
• Forgetting to classify errors: If you retry on every error equally, you'll waste time and resources. Spend time mapping your error types to retriable vs. permanent categories.
• No monitoring on DLQ depth: A growing DLQ is a symptom of a systemic issue. Monitor DLQ lag and alert if it exceeds a threshold (e.g., more than 100 messages in the last hour).
• Losing context in retry messages: Always include the original event, error details, and retry count when moving to a retry topic. Without this, you can't debug or replay effectively.

Key takeaways

• Retry topics and dead-letter queues decouple failure handling from the main partition, preventing one problematic message from blocking thousands behind it.
• Use a tiered retry approach (1m, 10m, DLQ) rather than exponential backoff in place—it maintains throughput and simplifies code.
• Classify errors explicitly: transient errors retry, permanent errors go straight to the DLQ.
• Make DLQs actionable by storing full context and providing a replay tool for operators to recover after underlying issues are fixed.
• Monitor DLQ depth and set alerts—a growing DLQ signals a production issue that needs immediate attention.
• Cap retry attempts at a reasonable number (2–3) to prevent message explosion and resource exhaustion.