Real-time metrics to track: patterns, pitfalls and fixes

Introduction

After 89 days of building real-time systems with Kafka and WebSockets, you've learned the architecture, the pitfalls, and how to scale. But without the right metrics, you're flying blind. This article covers real-time metrics to track: patterns, pitfalls and fixes—five numbers that belong on every production dashboard and why they matter more than you think.

We'll walk through each metric with concrete scenarios you've probably encountered: a spike in reconnects at 3 AM, a mysterious tail latency that only happens on Fridays, or connections that leak memory week after week. By the end, you'll know exactly what to instrument, where to look when things break, and how to read the signals before your users see the problem.

Active Connections

Active connections is the heartbeat of any real-time system. It tells you how many clients are currently online, and tracking its shape—not just its value—reveals everything from deployment issues to gradual memory leaks.

In a typical SaaS dashboard, active connections should follow a predictable daily curve: climb during business hours, dip overnight, and repeat. When it doesn't, something's wrong. A sudden cliff-drop means a deployment went sideways. A slow bleed over days means connections aren't being cleaned up properly. A saw-tooth pattern every 5 minutes suggests your load balancer is cycling servers or your heartbeat logic is broken.

Here's how to emit this metric from a NestJS WebSocket gateway:

import { WebSocketGateway, OnGatewayConnection, OnGatewayDisconnect } from '@nestjs/websockets';
import { Socket } from 'socket.io';
import { MetricsService } from './metrics.service';

@WebSocketGateway({ namespace: 'live' })
export class LiveGateway implements OnGatewayConnection, OnGatewayDisconnect {
  private activeConnections = new Set<string>();

  constructor(private metricsService: MetricsService) {}

  handleConnection(socket: Socket) {
    this.activeConnections.add(socket.id);
    this.metricsService.gauge('ws.connections.active', this.activeConnections.size);
  }

  handleDisconnect(socket: Socket) {
    this.activeConnections.delete(socket.id);
    this.metricsService.gauge('ws.connections.active', this.activeConnections.size);
  }
}

Plot this on a time-series graph with a 1-minute granularity. Alert if it drops by more than 20% in a 2-minute window—that's usually a deployment or network event you need to know about. Track percentiles too: p95 and p99 active connections tell you about your peak load and whether your capacity planning is realistic.

Messages Per Second

Messages per second (MPS) is your throughput gauge. It's not just about knowing you're handling 10k MPS—it's about knowing when that number changes and why.

In a trading platform, MPS spikes during market open and close. In a ride-hailing app, it spikes when surge pricing kicks in. If your MPS drops during a time when it should climb, you've got a bottleneck upstream: maybe Kafka is lagging, maybe your consumer is blocked, or maybe clients stopped sending. If MPS climbs unexpectedly, check for duplicate sends (a common client-side bug) or a runaway loop in your message processor.

Instrument both inbound (from clients to server) and outbound (server to clients) MPS separately. They should track closely. If outbound is much lower than inbound, you're either filtering heavily or dropping messages.

export class KafkaConsumerService {
  private messageCount = 0;
  private lastReportTime = Date.now();

  async processMessage(message: KafkaMessage) {
    this.messageCount++;

    const now = Date.now();
    if (now - this.lastReportTime >= 1000) {
      const mps = this.messageCount / ((now - this.lastReportTime) / 1000);
      this.metricsService.gauge('kafka.messages_per_second', mps);
      this.messageCount = 0;
      this.lastReportTime = now;
    }

    // Process the message
    await this.handleMessage(message);
  }
}

Use a histogram to track MPS over time, and alert on two thresholds: a lower bound (if it drops below your 95th percentile baseline) and an upper bound (if it exceeds your expected peak by 50%). This catches both consumer lag and runaway producers.

End-to-End Latency

End-to-end latency is where the rubber meets the road. A client sends a message, it flows through Kafka, gets processed, and bounces back. How long does that take? P50, P95, and P99 all tell different stories.

In an e-commerce checkout flow, latency under 100ms feels instant. Latency over 500ms feels stuck. In a trading system, 10ms latency is normal; 50ms latency might cost you money. The point: you need to know your baseline and detect when you're drifting.

Measure this end-to-end by adding a timestamp to every message on the client side, then recording how long it took to round-trip:

// Client side (React/Next.js)
const sendMessage = (payload: any) => {
  const sentAt = performance.now();
  socket.emit('request', { payload, sentAt }, (response) => {
    const roundTripMs = performance.now() - sentAt;
    analytics.trackMetric('rtt_ms', roundTripMs);
  });
};

// Server side
socket.on('request', ({ payload, sentAt }, callback) => {
  // Process payload
  const result = await processPayload(payload);
  
  // Client receives this and calculates round-trip
  callback({ result, sentAt });
});

Watch for creeping p99 latency—it often precedes a full outage. If p99 jumps from 50ms to 200ms, your system is under stress. Check Kafka offset lag, consumer thread pool saturation, and database query times. These are almost always the culprit.

Reconnect Rate

Reconnect rate tells you how many times per hour clients are losing connection and re-establishing. A healthy system should have a low, stable baseline. A spike means network problems, server issues, or a client-side bug that's causing rapid disconnects.

We've seen reconnect rates jump from 2 per minute to 20 per minute after a single deploy. We've also seen them creep up over days as a load balancer's connection timeout drifted out of sync with client keepalive logic. The metric catches both.

Instrument this in two places: on the server, when you see a reconnection event; and on the client, when it detects a disconnect and initiates reconnect:

// Server: track reconnections per client
socket.on('reconnect', () => {
  const clientId = socket.handshake.query.clientId;
  this.metricsService.increment('ws.reconnects', { client: clientId });
  this.metricsService.increment('ws.reconnects_total');
});

// Client: track from the client perspective
socket.on('disconnect', () => {
  console.log('Disconnected, will retry...');
});

socket.on('reconnect_attempt', () => {
  analytics.trackEvent('reconnect_attempt');
});

socket.on('reconnect', () => {
  analytics.trackEvent('reconnect_success');
});

Create a dashboard that shows reconnect rate as a time series and break it down by client version, region, or network type if possible. A spike in a specific region suggests an ISP or regional issue. A spike after a deploy suggests your server changes broke something.

Buffered Bytes Per Connection

Buffered bytes per connection is the sneaky metric that catches memory leaks and backpressure problems before they explode. When a client is slow to consume messages, your server buffers them in memory. If this buffer grows without bound, you'll eventually run out of RAM and crash.

In a scenario we saw in production: clients on slow 4G networks couldn't keep up with a 5 MPS stream. The server buffered messages, memory usage climbed, and after 20 minutes the process was killed by the OOM killer. We never would've caught it without buffered bytes per connection.

Track this by monitoring the socket's internal buffer:

@WebSocketGateway()
export class LiveGateway {
  private connectionMetrics = new Map<string, any>();

  @SubscribeMessage('subscribe')
  handleSubscribe(socket: Socket, payload: any) {
    const clientId = socket.id;
    const stream = this.getStream(payload);

    // Emit messages and track buffer size
    stream.on('message', (msg) => {
      const bufferedBytes = socket.outputBuffer?.length || 0;
      this.metricsService.gauge('ws.buffer_bytes', bufferedBytes, { client: clientId });

      socket.emit('message', msg);

      // Alert if buffer is growing too fast
      if (bufferedBytes > 1024 * 1024) { // 1 MB
        this.logger.warn(`High buffer for ${clientId}: ${bufferedBytes} bytes`);
      }
    });
  }
}

This metric is especially important if you're streaming high-frequency data. Plot it as a heatmap or histogram—most connections should have near-zero buffer, but a few slow ones will push the p99 higher. If p99 exceeds a few MB, you need backpressure logic: slow down or drop messages for slow consumers.

Common Mistakes

Not tracking percentiles: Average active connections of 1,000 means nothing if p99 is 10,000. Always track p50, p95, and p99.

Mixing inbound and outbound: Treating "messages per second" as a single number hides the direction. Track inbound and outbound separately, and alert if they diverge.

Ignoring the shape of the curve: A metric's value matters, but its shape over time matters more. A gradual climb means one problem. A sudden cliff means another. Build dashboards that show trend, not just current state.

Setting alerts at the wrong threshold: Alert thresholds should be based on your baseline, not on arbitrary numbers. Use the 95th or 99th percentile of normal behavior as your alert boundary, and add 20-30% headroom.

Not correlating metrics: When something breaks, all five metrics usually move at once. A reconnect spike often comes with a latency spike and a drop in active connections. Use a good observability platform that lets you see these together.

Key Takeaways

• Active connections reveal deployment problems and gradual leaks; track its shape, not just its value.
• Messages per second should be measured separately for inbound and outbound; a divergence indicates filtering, dropping, or a bottleneck.
• End-to-end latency is your user-facing metric; creeping p99 latency often precedes an outage.
• Reconnect rate spikes are early signals of network, server, or client-side issues; investigate immediately.
• Buffered bytes per connection catches memory leaks and backpressure problems before they become crashes.
• Always track percentiles (p50, p95, p99), not just averages; set alerts based on your baseline, not arbitrary numbers.
• Correlate metrics across your dashboard; when one metric moves, the others usually follow.