100 days of Kafka and WebSockets: lessons for production

I shipped my first real-time system exactly 100 days ago, and I've learned more about failure modes in production than in the five years before that. This post wraps up 100 days of Kafka and WebSockets: lessons learned building platforms where data freshness matters, where users feel stale information immediately, and where a single misconfiguration cascades into alerts at 2 AM.

Reliability is designed in — Idempotency, ordering and retries are architectural choices

When you start with WebSockets, your first instinct is to treat every message as a fire-and-forget event. That breaks catastrophically the moment you have network hiccups, connection drops, or a consumer that processes the same message twice.

I learned this the hard way in a trading platform where duplicate executions cost real money. The problem wasn't the WebSocket layer—it was that I'd assumed messages would arrive exactly once. They don't. Kafka guarantees at-least-once delivery, which means your consumer must be idempotent.

Idempotency starts with a unique identifier for every operation. For financial trades, we use a combination of user ID, order timestamp, and a client-generated request ID. The key is storing this in your database as a unique constraint, so replaying the same message becomes a no-op.

// Idempotent trade execution
async function executeTrade(trade: Trade) {
  const existingTrade = await db.trades.findUnique({
    where: { 
      requestId: trade.requestId 
    }
  });

  // Already processed this exact trade
  if (existingTrade) {
    return existingTrade;
  }

  // First time seeing this trade
  const executed = await db.trades.create({
    data: {
      requestId: trade.requestId,
      userId: trade.userId,
      symbol: trade.symbol,
      quantity: trade.quantity,
      executedAt: new Date()
    }
  });

  return executed;
}

Ordering matters just as much. If you send "create order" then "cancel order" into Kafka, and they arrive at the consumer in reverse order, you've deleted an order that never existed. Kafka guarantees ordering per partition, which means you must choose your partition key carefully. We use user ID as the partition key for user-specific events—this ensures all events for one user flow through the same partition in order.

Retries are not just about persistence. They're about your retry strategy. Exponential backoff with jitter prevents thundering herds. We log every retry with timestamp and error reason. When something breaks at 2 AM, those logs tell you whether the system recovered on its own or if you need to wake up.

// Retry with exponential backoff
async function sendToKafkaWithRetry(message: Message, maxRetries = 3) {
  for (let attempt = 0; attempt <= maxRetries; attempt++) {
    try {
      await producer.send({
        topic: 'trades',
        messages: [{ key: message.userId, value: JSON.stringify(message) }]
      });
      return;
    } catch (error) {
      if (attempt === maxRetries) throw error;
      
      const delayMs = Math.pow(2, attempt) * 1000 + Math.random() * 1000;
      console.log(`Retry attempt ${attempt + 1} after ${delayMs}ms`, error.message);
      await new Promise(r => setTimeout(r, delayMs));
    }
  }
}

The lesson: reliability isn't something you add later. It lives in your schema (unique constraints), your message design (idempotency keys), and your infrastructure (partition keys, retry logic). Build it from day one.

Simple transports first — Most features never needed WebSockets

Here's a hard truth: I started with WebSockets for everything because I thought real-time sounded cool. I shipped exactly three features that genuinely needed WebSockets. The rest would have worked fine with polling or simple HTTP endpoints.

WebSockets have real costs. They keep connections alive, they consume memory, they complicate your deployment (load balancers need sticky sessions or you need connection affinity), and they fail silently if your browser doesn't handle reconnection logic properly. For a ride-hailing platform, we built real-time driver location updates with WebSockets. For the payment confirmation screen? Polling every 500ms on HTTPS was simpler, cheaper, and shipped faster.

The features that genuinely needed WebSockets were:

• Live notifications — Users see new messages, chat replies, or order status changes instantly. Polling feels sluggish and wastes bandwidth.
• Collaborative updates — Multiple users editing the same document, form, or configuration. You need to push conflicts to all clients immediately.
• Streaming data feeds — Stock prices, real-time dashboards, live metrics. These update frequently enough that polling becomes wasteful.

Everything else? HTTP with proper caching headers, or polling with reasonable intervals (2–5 seconds for most business features). A CRM system showing leads, deals, or contact lists doesn't need WebSockets. A user waiting for their password reset email doesn't need WebSockets.

I measured this on a SaaS platform. We had WebSocket connections for six different feature areas. Four of them sat idle 95% of the time. We moved four of them to polling, cut WebSocket connections by 66%, and reduced memory usage on our servers by 40%. The user experience didn't change at all.

Choose your transport based on actual user experience requirements, not on whether the technology is cool. Start with simple HTTP. Add WebSockets only when polling becomes unacceptable.

Observability is mandatory — Lag and latency dashboards save hours of debugging

The first time I deployed a Kafka consumer to production without proper observability, I discovered a bug 14 hours later when users started complaining about missing data. The consumer was crashing silently, restarting, and never catching up. Nobody knew because I wasn't monitoring consumer lag.

Consumer lag is the gap between the latest message in Kafka and the last message your consumer processed. High lag means your system is falling behind. It's the single most important metric for any real-time system.

We use Prometheus + Grafana with client libraries that expose these metrics automatically. For Node.js and Kafka, the KafkaJS client library exports metrics, or you can publish them manually.

// Expose Kafka consumer metrics to Prometheus
import { register, Gauge, Counter, Histogram } from 'prom-client';

const consumerLag = new Gauge({
  name: 'kafka_consumer_lag',
  help: 'Lag of Kafka consumer in messages',
  labelNames: ['topic', 'partition', 'group']
});

const messageProcessingDuration = new Histogram({
  name: 'kafka_message_processing_seconds',
  help: 'Time to process a Kafka message',
  labelNames: ['topic']
});

const processingErrors = new Counter({
  name: 'kafka_processing_errors_total',
  help: 'Total errors processing messages',
  labelNames: ['topic', 'error_type']
});

// In your consumer
consumer.on('message', async (message) => {
  const timer = messageProcessingDuration.startTimer({ topic: message.topic });
  try {
    await processMessage(message);
    timer();
  } catch (error) {
    processingErrors.inc({ topic: message.topic, error_type: error.name });
    timer();
    throw error;
  }
});

Latency dashboards are equally critical. We track end-to-end latency: time from when a user performs an action to when all connected WebSocket clients see the result. This tells you whether your system feels "real-time" to users. A 2-second latency feels broken, even if it's technically working.

The other critical observation: alert on lag, not on errors. A consumer that crashes and restarts can show a low error count but high lag. By the time errors spike, you're already 30 minutes behind. Lag catches problems earlier.

Real-time is a product experience — Users notice stale data more than slow pages

Real-time isn't about speed. It's about freshness. Users can tolerate a page taking 2 seconds to load. They cannot tolerate seeing yesterday's data on a dashboard that claims to be "live."

I learned this in a travel platform where we showed real-time seat availability. We added caching to make pages faster, but caching stale seat data led to double-booking. The product team wanted the page to load in 500ms. I wanted accurate data. We compromised: 500ms with potentially stale data, plus a visible disclaimer "as of 30 seconds ago" and a refresh button.

The real win? We showed "last updated" timestamps on every real-time component. Users could see that data was fresh. They tolerated slower updates if they knew how fresh the data was. The feature felt more real-time because it was honest about freshness.

This is where WebSockets shine: you push data to clients the moment it changes, plus you show the timestamp. Users see "updated 2 seconds ago" and trust that the information is current. Compare that to a page they refresh manually every 5 minutes.

// Send data with freshness metadata
const seatUpdate = {
  seats: availableSeats,
  updatedAt: new Date().toISOString(),
  isLive: true,
  refreshedAgo_ms: Date.now() - lastUpdateTime
};

ws.send(JSON.stringify(seatUpdate));

The product implication is profound. Real-time features compete not on speed but on trust. If your live dashboard shows data from 5 minutes ago with no indication, users will assume it's broken and stop trusting it. If it shows "live" with a timestamp, users accept even longer latencies because they understand the freshness.

Common mistakes

• Building WebSockets without a reconnection strategy — The moment a connection drops, clients hang silently. Implement exponential backoff reconnection, or you'll have phantom disconnects.
• Not deduplicating in the consumer — Assuming Kafka delivers each message exactly once is the fastest way to corrupt data. Always check your database for existing records before inserting.
• Monitoring message volume, not lag — A consumer can process millions of messages per second but still fall behind if the topic receives 10 million per second. Lag is the only metric that matters.
• Broadcasting to all clients instead of to relevant clients — Sending every update to every WebSocket connection doesn't scale. Implement filtering: subscribe users to specific topics (seats on route X, chat room Y, etc.).
• Ignoring ordering in multi-partition topics — If your consumer subscribes to multiple partitions, messages can arrive out of order. Use a timestamp or sequence number for reordering in your application layer.

Key takeaways

• Design reliability into your architecture with idempotency keys, unique constraints, and the right partition keys. You can't patch this in later.
• Use simple HTTP and polling for most features. WebSockets are for cases where freshness is a core product requirement, not for every feature that could theoretically benefit from it.
• Observability—specifically consumer lag and end-to-end latency—is not optional. Without it, bugs stay hidden for hours or days.
• Real-time is a trust property. Users believe in "live" data when you show them freshness. Hide the latency if you can, but never hide when data is stale.
• Partition by the right key. Choose user ID, room ID, or resource ID—anything that groups related messages together. This guarantees ordering for the cases where it matters.
• Test failure modes explicitly: consumer crashes, network partitions, duplicate messages. These are not edge cases; they're certainties in production.