My real-time systems checklist: best practices explained

Introduction

By day 98 of running real-time systems in production, you've learned that shipping fast means nothing if your feature breaks under load or loses data when a client reconnects. We've built trading dashboards that drop messages, e-commerce checkouts that stall mid-transaction, and ride-hailing services that can't sync rider location properly. My real-time systems checklist: best practices explained is what I now verify before shipping any live feature—because the gaps between "works on my machine" and "works for 50,000 concurrent users" are where careers get difficult.

This article walks through the exact checklist we run through in the final sprint before deployment. It's not theoretical; every item comes from a production incident we've debugged.

Transport — Simplest option chosen: polling, SSE or WebSockets

The first decision shapes everything downstream. Too many teams default to WebSockets because they sound real-time, then spend months tuning connection limits. Sometimes polling is the right answer.

Ask yourself: How fresh must the data be? What's the user tolerance for latency? How many concurrent subscribers per server?

In a travel platform, flight price updates can tolerate 3–5 second delays, so we use HTTP long-polling with a 30-second timeout. Clients reconnect gracefully, infrastructure is simpler, and we avoid WebSocket connection exhaustion. The trade-off: slightly higher CPU from repeated requests, but that's accepted.

In a CRM where sales reps need instant notification of lead assignment, WebSockets make sense. We maintain persistent connections and broadcast assignment events through Kafka to all connected clients. The payload is small (a user ID and lead ID), so bandwidth is negligible.

Server-Sent Events (SSE) sits in the middle: one-way push, HTTP-based, works behind most proxies. Use it for notification feeds or status streams where you don't need client-to-server messaging.

Checklist item: Have we benchmarked the transport choice against actual expected concurrency? Does polling add unacceptable latency or CPU? Does WebSockets fit within our infrastructure's connection budget?

Reliability — Reconnection, resync and idempotent handling

Assuming you choose WebSockets, the real work begins. A client loses network for 2 seconds. What state is it in now? Has the server cleaned up its connection? Did the client miss an update?

We structure every real-time feature around three layers: message acknowledgment, client state tracking, and server state verification.

// Client-side reconnection with exponential backoff
class RealtimeClient {
  private ws: WebSocket | null = null;
  private reconnectAttempts = 0;
  private maxReconnectAttempts = 5;
  private lastMessageId = 0;

  connect() {
    this.ws = new WebSocket(this.url);
    
    this.ws.onopen = () => {
      this.reconnectAttempts = 0;
      // Resync: ask server for all state since lastMessageId
      this.sendMessage({
        type: 'SYNC_REQUEST',
        since: this.lastMessageId
      });
    };

    this.ws.onmessage = (event) => {
      const msg = JSON.parse(event.data);
      this.lastMessageId = msg.id;
      this.handleMessage(msg);
      // Send ack back to server
      this.sendMessage({ type: 'ACK', messageId: msg.id });
    };

    this.ws.onclose = () => this.reconnect();
  }

  private reconnect() {
    if (this.reconnectAttempts >= this.maxReconnectAttempts) {
      this.emit('error', 'Max reconnection attempts reached');
      return;
    }
    const delay = Math.pow(2, this.reconnectAttempts) * 1000;
    this.reconnectAttempts++;
    setTimeout(() => this.connect(), delay);
  }
}

On the server side, we track which clients have acknowledged which message IDs. If a client reconnects, we replay any unacked messages from our buffer (typically 100–500 messages, stored in Redis).

// Server-side message tracking
class MessageBuffer {
  private buffer: Map = new Map();
  private maxBufferSize = 500;

  addMessage(clientId: string, msg: any) {
    if (!this.buffer.has(clientId)) {
      this.buffer.set(clientId, []);
    }
    const log = this.buffer.get(clientId)!;
    log.push({ id: log.length + 1, data: msg, timestamp: Date.now() });
    if (log.length > this.maxBufferSize) {
      log.shift();
    }
  }

  getUnackedMessages(clientId: string, sinceId: number) {
    const log = this.buffer.get(clientId) || [];
    return log.filter(m => m.id > sinceId);
  }

  acknowledgeMessage(clientId: string, messageId: number) {
    // Remove acked message from buffer (optional; depends on retention policy)
  }
}

Critical: Ensure every operation is idempotent. If a client receives the same update twice (due to resync or replay), it shouldn't apply the change twice. Use version numbers or timestamps to deduplicate.

Checklist item: Can a client reconnect and recover its state without losing data or duplicating updates? Is message replay tested under network flakiness? Are operations idempotent?

Scale — Backplane, connection limits and load tests

One server handles 10,000 WebSocket connections comfortably. What happens when you need 100,000? Enter the backplane.

We use Kafka as our message bus and Redis for connection registry. When a client connects to Server A, we register it in Redis. When we need to broadcast an event (e.g., "order shipped"), we publish to Kafka, and every server subscribes and pushes to connected clients.

// Kafka consumer on each server
const consumer = kafka.consumer({ groupId: 'realtime-servers' });
await consumer.subscribe({ topic: 'order-events' });

await consumer.run({
  eachMessage: async ({ partition, message }) => {
    const event = JSON.parse(message.value.toString());
    const connectedClients = await redis.get(`subscribers:order:${event.orderId}`);
    
    connectedClients.forEach(clientId => {
      const ws = connectionMap.get(clientId);
      if (ws && ws.readyState === WebSocket.OPEN) {
        ws.send(JSON.stringify({ type: 'ORDER_SHIPPED', ...event }));
      }
    });
  }
});

Redis also tracks which server each client is connected to, so targeted messaging is possible (e.g., "notify only this user").

Connection limits: Set realistic ulimits on your servers. A single Node.js process peaks around 15,000–20,000 concurrent connections before memory and CPU become issues. Load-test aggressively with a tool like Artillery or Locust.

Checklist item: Have we load-tested to 2x our projected peak concurrency? Does the backplane (Kafka + Redis) scale horizontally? Are connection limits documented and monitored?

Security — Auth on connect, authorization per action, rate limits

WebSocket connections bypass traditional HTTP middleware. Authentication and authorization must happen differently.

On connect: Verify the JWT or session token before upgrading to WebSocket. Reject unauthenticated connections immediately.

// Express + WebSocket upgrade handler
io.use((socket, next) => {
  const token = socket.handshake.auth.token;
  if (!token) {
    return next(new Error('Authentication error'));
  }
  try {
    const user = jwt.verify(token, process.env.JWT_SECRET);
    socket.data.userId = user.id;
    socket.data.permissions = user.permissions;
    next();
  } catch (err) {
    next(new Error('Invalid token'));
  }
});

Per-action authorization: Even after connecting, validate that the user can perform the requested action. Don't assume their permissions didn't change since login.

socket.on('subscribe_to_order', async (orderId) => {
  const canAccess = await checkOrderAccess(socket.data.userId, orderId);
  if (!canAccess) {
    socket.emit('error', { message: 'Unauthorized' });
    return;
  }
  socket.join(`order:${orderId}`);
});

Rate limiting: Apply per-user and per-connection rate limits. A malicious or buggy client shouldn't DoS the service.

Checklist item: Is authentication enforced on WebSocket upgrade? Are permissions re-validated per action? Are rate limits in place? Is token refresh handled gracefully?

Observability — Connections, latency and lag dashboards

If you can't see it, you can't debug it. Real-time systems fail silently: a connection stays open but stops receiving messages. Your dashboard must surface this.

Key metrics:

• Active connections per server: Alert if any server drops suddenly.
• Message latency (p50, p95, p99): Time from Kafka publish to client receive. Catch backlog buildup.
• Lag: How many messages are queued for dispatch. High lag = slow consumers or downstream bottleneck.
• Reconnection rate: Spike in reconnects often signals network issues or server restarts.
• Message loss rate: Track how many resync requests occur. High rate suggests buffer overflow.

We export these via Prometheus and visualize in Grafana. Alerts fire when reconnection rate exceeds 5% of connections or message latency p99 exceeds 500ms.

// Prometheus metrics
const activeConnections = new prometheus.Gauge({
  name: 'realtime_active_connections',
  help: 'Number of active WebSocket connections',
  labelNames: ['server_id']
});

const messageLatency = new prometheus.Histogram({
  name: 'realtime_message_latency_ms',
  help: 'Latency from publish to delivery',
  buckets: [10, 50, 100, 250, 500, 1000, 5000]
});

// In your message handler
const startTime = Date.now();
ws.send(JSON.stringify(message));
messageLatency.observe(Date.now() - startTime);

Checklist item: Do we have dashboards for connection count, message latency, and lag? Are alerts configured for anomalies? Can we correlate metrics with deployment events?

Common mistakes

• Ignoring the network layer: Assuming WebSocket connections never drop. They do, constantly, in production. Test with tc (traffic control) to inject latency and packet loss.
• Overengineering transport: Starting with WebSockets when polling would suffice. Complexity breeds bugs.
• No idempotency: Broadcasting the same message twice during resync causes duplicate charges, duplicate notifications, or corrupted state.
• Insufficient backpressure handling: If a client can't keep up, the server buffers indefinitely, leaking memory. Implement a watermark: if queue depth exceeds 1000 messages, disconnect and force reconnect.
• Missing reconnection tests: Your test suite passes, but behavior under network partition fails in production. Simulate disconnections explicitly.
• Forgetting to refresh tokens: WebSocket connections live for hours. If your JWT expires after 1 hour, the user is silently disconnected mid-session. Implement token refresh over the WebSocket itself.

Key takeaways

• Choose transport based on latency tolerance and concurrency: polling for loose coupling, WebSockets for true real-time, SSE for one-way push.
• Build reconnection logic from day one: exponential backoff, server-side message buffering, idempotent operations.
• Scale with a backplane (Kafka + Redis or similar): one server shouldn't handle all connections and broadcasts.
• Authenticate on connect, authorize per action, and rate-limit aggressively.
• Instrument everything: track connections, latency, lag, and reconnect rates. Blind spots hide production incidents.
• Test under realistic network conditions (packet loss, jitter, disconnections) before shipping.