Why every senior engineer should know events and real-time

You're debugging a ride-tracking feature at 2 AM when orders start disappearing from the driver's queue. Nobody's seeing real-time updates on the app, and your support team is flooded. After an hour, you realize the WebSocket connection died silently, but the backend kept publishing events to Kafka with nobody listening. This is why every senior engineer should know events and real-time—because getting them right keeps systems alive, and getting them wrong breaks everything users depend on.

The gap between moving data and showing it

When I first joined a ride-hailing platform, I made a classic mistake: I thought moving data between services and moving updates to users were the same problem. They're not. I could ship an event to Kafka perfectly—reliable, ordered, persisted—but the user's app would still show stale information. The reverse happened too: WebSocket updates would arrive instantly but disappear if the backend crashed, leaving the frontend out of sync forever.

That gap is where most real-time systems fail silently. Senior engineers need to understand that events and real-time solve different problems. Events are about reliable, ordered delivery between services. Real-time is about instant delivery to users. You need both, and they talk to each other in ways that aren't obvious until they break.

In a ride-hailing system, a driver's location changes thousands of times per second. Kafka captures those events reliably between the location service, matching service, and analytics—nothing gets lost. WebSockets push the latest location to the rider's phone instantly—sub-100ms ideally. One failure kills reliability, the other kills user experience. Get both wrong and you get neither.

Kafka moves events reliably between services

Kafka is your source of truth for what happened, and when. Every event that matters—an order placed, a payment processed, a driver matched—goes into Kafka with a timestamp, partition, and offset. Services consume at their own pace. If a consumer crashes, it picks up where it left off. If you need to replay all events from yesterday to debug something, you can.

In the trading platform I worked on, every price tick, every trade, every position update went to Kafka. The order service consumed trades, the reporting service consumed trades, the risk service consumed trades, all independently. When reporting fell behind during market open, it didn't block trading. When we needed to recalculate P&L for a week, we replayed the topic. That's the power of events as an event stream.

Here's a minimal producer example in NestJS:

import { Injectable } from '@nestjs/common';
import { Kafka, logLevel } from 'kafkajs';

@Injectable()
export class OrderEventProducer {
  private kafka = new Kafka({
    clientId: 'order-service',
    brokers: ['localhost:9092'],
    logLevel: logLevel.ERROR,
  });
  
  private producer = this.kafka.producer();

  async onModuleInit() {
    await this.producer.connect();
  }

  async publishOrderCreated(orderId: string, customerId: string, amount: number) {
    await this.producer.send({
      topic: 'orders.created',
      messages: [
        {
          key: orderId,
          value: JSON.stringify({
            orderId,
            customerId,
            amount,
            timestamp: Date.now(),
            eventType: 'order.created',
          }),
        },
      ],
    });
  }

  async onModuleDestroy() {
    await this.producer.disconnect();
  }
}

Notice the key: orderId. Kafka partitions on key, so all events for one order go to the same partition, preserving order. That's critical. Without it, your consumer sees events out of order and your state gets corrupted.

A consumer looks like this:

@Injectable()
export class OrderEventConsumer {
  private kafka = new Kafka({
    clientId: 'notification-service',
    brokers: ['localhost:9092'],
  });

  async onModuleInit() {
    const consumer = this.kafka.consumer({ groupId: 'notifications' });
    await consumer.connect();
    await consumer.subscribe({ topic: 'orders.created', fromBeginning: false });
    
    await consumer.run({
      eachMessage: async ({ topic, partition, message }) => {
        const event = JSON.parse(message.value.toString());
        await this.sendOrderConfirmationEmail(event.customerId, event.orderId);
      },
    });
  }
}

The consumer group tracks offset. If it crashes mid-email, it resumes from the last committed offset. You won't resend the same email, but you won't miss one either. That's the reliability guarantee that makes Kafka trustworthy for events.

WebSockets move updates instantly to users

Kafka is great for durability. It's terrible for latency. By the time an event is in Kafka, marshalled, sent to a consumer, processed, and ready to display, 50-500ms has passed. For ride tracking, that's a frozen map. For a trading screen, that's a missed opportunity. For CRM alerts, that's annoying lag.

WebSockets solve this. They're persistent, bidirectional connections where the server pushes data instantly. No polling, no request-response cycle, no delay. A price update hits Kafka, a processor grabs it, and within 10-50ms it's on the trader's screen.

Here's a NestJS WebSocket setup with a real-time order status feed:

import { WebSocketGateway, WebSocketServer, SubscribeMessage } from '@nestjs/websockets';
import { Server, Socket } from 'socket.io';

@WebSocketGateway({
  namespace: '/orders',
  cors: { origin: '*' },
})
export class OrderStatusGateway {
  @WebSocketServer()
  server: Server;

  @SubscribeMessage('watch_order')
  handleWatchOrder(client: Socket, orderId: string) {
    client.join(`order:${orderId}`);
    client.emit('watching', { orderId, timestamp: Date.now() });
  }

  // Called by the order service when status changes
  emitOrderStatusChange(orderId: string, status: string, details: any) {
    this.server.to(`order:${orderId}`).emit('order_updated', {
      orderId,
      status,
      details,
      timestamp: Date.now(),
    });
  }
}

On the frontend (React/Next.js):

import { useEffect, useState } from 'react';
import { io } from 'socket.io-client';

export function OrderTracker({ orderId }) {
  const [status, setStatus] = useState(null);
  const [socket, setSocket] = useState(null);

  useEffect(() => {
    const s = io('http://backend:3000/orders');
    s.emit('watch_order', orderId);
    
    s.on('order_updated', (data) => {
      setStatus(data.status);
    });

    setSocket(s);
    return () => s.disconnect();
  }, [orderId]);

  return <div>Status: {status}</div>;
}

The user sees status updates live. But here's the catch: if the WebSocket disconnects, the client doesn't know. It keeps showing "Pending" even if the order shipped. That's why you need Kafka behind it—a source of truth the WebSocket can query on reconnect, and a queue of missed events the server can send when the client comes back online.

Together they power most 'live' features

The real-time stack is this: Kafka is the spine, WebSockets are the nerves. The order service publishes to Kafka. A processor reads from Kafka, updates a cache (Redis), and signals the WebSocket gateway via an in-memory queue or direct call. The gateway pushes to connected clients. If a client disconnects, the gateway gracefully closes the subscription. On reconnect, the client queries the latest state from the API (which read from Redis/cache), then re-subscribes to live updates.

In the ride-hailing app I built, a rider subscribes to driver location updates via WebSocket. The driver's location service streams to Kafka every 2 seconds. A real-time aggregator consumes Kafka, deduplicates and compresses positions, and broadcasts to WebSocket rooms (all users watching that trip). The aggregator runs in parallel—Kafka ensures no locations are lost even if the aggregator is slow. WebSockets ensure the UI updates instantly.

You need all four layers. Miss any and real-time breaks in subtle ways.

Getting them wrong causes the hardest bugs

I've spent weeks debugging bugs that only happened when Kafka was fast but WebSockets were slow. Picture this: a CRM agent updates a contact's phone number. The event goes to Kafka instantly, a webhook consumer updates the database, but the WebSocket gateway is swamped and takes 5 seconds to push the update. Another agent on the same contact reloads the page (because they didn't see the update live) and overwrites it with stale data. Now you have a corrupt record and no audit trail of how it happened.

Another classic: WebSocket works great in development, but at scale (10K concurrent users), the single server can't keep up. Events pile up in memory, the server runs out of RAM, crashes, and loses all subscriptions. Users see "Connection lost" and manually refresh, creating a thundering herd that crashes the API.

The hardest bugs I've hit involve Kafka and WebSockets being out of sync:

• Out-of-order delivery: A payment is confirmed, then refunded. If the WebSocket pushes refund first and Kafka delivers it second, the UI shows a credit that later disappears. The user panics.
• Silent reconnect failures: A WebSocket reconnects but doesn't resub scribe to its topics. It looks connected but receives no updates. Takes hours to notice because tests don't catch it.
• Kafka replay with live subscribers: You replay Kafka to backfill missing data, but live WebSocket subscribers see historical events as live. User thinks their order was cancelled today when it actually failed last week.
• Partition rebalancing hangs: Kafka consumer group rebalances, stops processing for 30 seconds. WebSocket has no fallback, users see frozen screens, support blows up.

These aren't theoretical. I've seen all of them in production. Senior engineers need to know events and real-time because debugging these requires understanding both systems deeply—what guarantees each makes, where they can diverge, and how to bridge the gap.

Common mistakes

• Treating WebSocket as durable storage: Don't store important state only in WebSocket memory. Always back it with Kafka and a database.
• Ignoring partitioning: If you don't partition Kafka topics by a meaningful key (orderId, userId), events can get out of order. State machines break silently.
• No reconnection logic: When a WebSocket drops, the client must detect it, reconnect, and resync state. Most teams forget this, causing stale UIs.
• Assuming WebSocket scales linearly: Each connection is a file descriptor and memory. At 100K concurrent users on one server, you'll hit the limits. Design for horizontal scaling from day one (rooms, load balancing).
• Not instrumenting latency: You can't optimize what you don't measure. Instrument: event publishing latency, Kafka end-to-end latency, WebSocket push latency. Build dashboards.
• Mixing temporal and causal ordering: Kafka guarantees causal order within a partition (same key). But users might be watching multiple partitions. The UI might show events out of their true temporal order. Timestamp every event and reconcile on the client.

Key takeaways

• Kafka guarantees durable, ordered event delivery between services. WebSockets guarantee instant delivery to users. They solve different problems and both are necessary.
• Kafka partitioning on the right key (orderId, userId) is critical for preserving order. Don't skip this.
• WebSocket reconnection and resync logic is easy to miss and causes silent failures. Build it explicitly and test it.
• The real-time stack is: Kafka (durability) → Cache (current state) → WebSocket (push) → REST (fallback). Miss any layer and you'll regret it.
• Out-of-order events, silent disconnects, and partition rebalancing are the bugs that hurt most. Know how to detect and recover from each.
• Instrument latency across the entire pipeline. If events take 2 seconds to reach users, your system isn't real-time.