You're running a production ride-hailing system where drivers and passengers connect via WebSocket, and suddenly your support team reports that some users can't see real-time location updates. You log into your dashboard, but your monitoring dashboards only show CPU and memory—nothing about what's actually happening inside those WebSocket connections. This is where observability for WebSockets becomes critical: understanding the health and performance of your real-time systems means measuring what matters at the connection, message, and latency levels.
Building production WebSocket systems is fundamentally different from stateless HTTP services. In my experience scaling real-time platforms—from e-commerce notifications to ride-hailing location streams—I've learned that traditional infrastructure metrics miss the story entirely. You need to measure the right signals at the real-time layer to catch issues before they cascade into user-facing outages.
Connections—The Foundation of Real-Time Health
The first observable metric in any WebSocket system is connection churn: how many clients are connecting and disconnecting, and how many are active right now. I track three key signals here.
Active connections tells you the current load on your WebSocket servers. If you're serving a ride-hailing app and active connections drop 30% in 5 minutes with no deployment, something broke. I expose this via Prometheus:
const activeConnections = new prometheus.Gauge({
name: 'websocket_active_connections',
help: 'Number of active WebSocket connections',
labelNames: ['namespace', 'server_id']
});
wss.on('connection', (ws) => {
activeConnections.inc({ namespace: 'ride_tracking', server_id: process.env.SERVER_ID });
ws.on('close', () => {
activeConnections.dec({ namespace: 'ride_tracking', server_id: process.env.SERVER_ID });
});
});Connects and disconnects per minute reveals churn patterns. A normal system has steady connection churn as clients reconnect, refresh browsers, or switch networks. Abnormal patterns—spikes in disconnects without corresponding connects—indicate network issues, server crashes, or auth token expiry. I measure this as a counter that increments on each event:
const connectionCounter = new prometheus.Counter({
name: 'websocket_connections_total',
help: 'Total WebSocket connection attempts',
labelNames: ['namespace', 'status'] // 'established' or 'failed'
});
const disconnectCounter = new prometheus.Counter({
name: 'websocket_disconnects_total',
help: 'Total WebSocket disconnections',
labelNames: ['namespace', 'reason'] // 'client_closed', 'server_closed', 'timeout'
});
wss.on('connection', (ws) => {
connectionCounter.inc({ namespace: 'ride_tracking', status: 'established' });
ws.on('close', (code, reason) => {
const reason_label = code === 1000 ? 'client_closed' : 'server_closed';
disconnectCounter.inc({ namespace: 'ride_tracking', reason: reason_label });
});
});In a real production scenario, I once debugged a mystery where connections spiked and dropped every 30 seconds. Turns out, a mobile client was reconnecting due to a DNS timeout. Without tracking these metrics, I would have spent hours looking at server logs instead of identifying the pattern immediately.
Messages—The Throughput Story
Connection counts tell you whether clients are connected; message metrics tell you whether they're actually exchanging data.
Messages in and out per second is your throughput indicator. I measure this separately for inbound (client to server) and outbound (server to client) traffic:
const messagesInCounter = new prometheus.Counter({
name: 'websocket_messages_in_total',
help: 'Total WebSocket messages received from clients',
labelNames: ['namespace', 'message_type']
});
const messagesOutCounter = new prometheus.Counter({
name: 'websocket_messages_out_total',
help: 'Total WebSocket messages sent to clients',
labelNames: ['namespace', 'message_type']
});
wss.on('connection', (ws) => {
ws.on('message', (data) => {
const msg = JSON.parse(data);
messagesInCounter.inc({
namespace: 'ride_tracking',
message_type: msg.type
});
// Process message...
broadcastUpdate(msg);
});
});
function broadcastUpdate(msg) {
wss.clients.forEach((client) => {
if (client.readyState === WebSocket.OPEN) {
client.send(JSON.stringify(msg));
messagesOutCounter.inc({
namespace: 'ride_tracking',
message_type: msg.type
});
}
});
}Payload sizes matter because bandwidth and serialization costs add up. A driver location update might be 150 bytes, but if you're broadcasting to 10,000 connected passengers, that's 1.5 MB per update. Track this as a histogram:
const payloadSize = new prometheus.Histogram({
name: 'websocket_payload_bytes',
help: 'WebSocket message payload size in bytes',
labelNames: ['namespace', 'direction'], // 'inbound' or 'outbound'
buckets: [50, 100, 500, 1000, 5000, 10000]
});
ws.on('message', (data) => {
payloadSize.observe({
namespace: 'ride_tracking',
direction: 'inbound'
}, Buffer.byteLength(data));
});In a CRM system I built, a dashboard refresh was sending 2 MB+ payloads per client due to unfiltered data. The histogram showed the outlier immediately, and we implemented delta encoding to reduce it to 50 KB. Without payload metrics, that inefficiency would have stayed hidden.
Latency—End-to-End Delivery Time
Low latency is the entire reason for WebSockets. Measuring it end-to-end—from event creation on the server to rendering on the client—is non-negotiable for real-time systems.
Time from event creation to client delivery is your core latency metric. I instrument this by adding a timestamp to each message:
const deliveryLatency = new prometheus.Histogram({
name: 'websocket_delivery_latency_ms',
help: 'Time from message creation to client delivery in milliseconds',
labelNames: ['namespace', 'message_type'],
buckets: [10, 50, 100, 250, 500, 1000, 2000]
});
function broadcastUpdate(msg) {
const createdAt = msg.timestamp || Date.now();
wss.clients.forEach((client) => {
if (client.readyState === WebSocket.OPEN) {
const payload = JSON.stringify(msg);
client.send(payload);
const latency = Date.now() - createdAt;
deliveryLatency.observe({
namespace: 'ride_tracking',
message_type: msg.type
}, latency);
}
});
}For a travel booking system, I noticed 95th percentile latency was spiking to 500ms+ during peak hours. Server-side, everything looked fine. Turns out, the mobile client was running on a poor network. We started measuring round-trip time (client sends ping, server echoes, client measures response) to separate network issues from server issues:
// Server-side: handle ping/pong for RTT measurement
ws.on('message', (data) => {
const msg = JSON.parse(data);
if (msg.type === 'ping') {
ws.send(JSON.stringify({
type: 'pong',
timestamp: msg.timestamp
}));
}
});
// Client-side: measure RTT
const rttLatency = new prometheus.Histogram({
name: 'websocket_rtt_ms',
help: 'Round-trip time in milliseconds',
buckets: [10, 50, 100, 250, 500, 1000]
});
function measureRTT() {
const start = Date.now();
ws.send(JSON.stringify({ type: 'ping', timestamp: Date.now() }));
ws.once('message', (data) => {
const msg = JSON.parse(data);
if (msg.type === 'pong') {
rttLatency.observe({}, Date.now() - start);
}
});
}This distinction helped us realize 70% of our latency was network-related, not server-related.
Errors—Finding Silent Failures
Errors in WebSocket systems are often silent. A message gets dropped, and neither client nor server reports it explicitly. Comprehensive error tracking prevents this blindness.
Auth failures should be tracked with labels for the failure reason:
const authFailures = new prometheus.Counter({
name: 'websocket_auth_failures_total',
help: 'WebSocket authentication failures',
labelNames: ['namespace', 'reason'] // 'invalid_token', 'expired_token', 'missing_auth'
});
wss.on('connection', (ws, req) => {
const token = extractToken(req);
if (!token) {
authFailures.inc({ namespace: 'ride_tracking', reason: 'missing_auth' });
ws.close(1008, 'Unauthorized');
return;
}
try {
const decoded = jwt.verify(token, process.env.SECRET);
} catch (err) {
const reason = err.name === 'TokenExpiredError' ? 'expired_token' : 'invalid_token';
authFailures.inc({ namespace: 'ride_tracking', reason });
ws.close(1008, 'Unauthorized');
}
});Dropped messages happen when the send buffer fills or the client disconnects mid-send. Track send failures:
const sendFailures = new prometheus.Counter({
name: 'websocket_send_failures_total',
help: 'Failed message sends',
labelNames: ['namespace', 'reason'] // 'closed', 'buffer_full'
});
function broadcastUpdate(msg) {
wss.clients.forEach((client) => {
if (client.readyState === WebSocket.OPEN) {
client.send(JSON.stringify(msg), (err) => {
if (err) {
sendFailures.inc({
namespace: 'ride_tracking',
reason: 'write_error'
});
}
});
} else {
sendFailures.inc({
namespace: 'ride_tracking',
reason: 'closed'
});
}
});
}Reconnection storms occur when many clients reconnect simultaneously, overwhelming your server. Track reconnection rate per client and aggregate:
const reconnectionRate = new prometheus.Gauge({
name: 'websocket_reconnect_attempts_per_client',
help: 'Reconnection attempts per client (exponential bucket)',
labelNames: ['namespace', 'bucket'] // bucket is time window
});
// Simple: count reconnects in 1-minute windows
const reconnectCounts = new Map();
function trackReconnect(clientId) {
if (!reconnectCounts.has(clientId)) {
reconnectCounts.set(clientId, 0);
}
reconnectCounts.set(clientId, reconnectCounts.get(clientId) + 1);
const count = reconnectCounts.get(clientId);
if (count > 10) {
console.warn(`Client ${clientId} reconnected ${count} times in 1 minute`);
}
}Common Mistakes
- Only measuring active connections. Connection count alone hides churn and error patterns. You need connects, disconnects, and reasons.
- Ignoring payload sizes. A system with low message throughput but huge payloads will hit bandwidth limits before you see message count alerts.
- Not separating client and server latency. Treating all latency as server latency leads to wasted optimization. Network RTT must be measured independently.
- Treating all disconnects equally. Client-initiated close (code 1000) is normal; server close or timeout is not. Label your metrics accordingly.
- Forgetting about message queuing latency. If you buffer messages before sending (e.g., batching), measure the time in queue separately from network transit time.
Key Takeaways
- Track active connections, connects/min, and disconnects/min to monitor connection health and detect churn anomalies.
- Measure messages in/out per second and payload sizes to understand throughput and identify serialization inefficiencies.
- Instrument end-to-end delivery latency and separate it from network latency using round-trip time (RTT) measurements.
- Monitor auth failures, dropped messages, and reconnection rates to catch silent errors before they impact users.
- Use consistent labels (namespace, message_type, reason) across all metrics so you can correlate signals during incidents.
- Set alert thresholds based on your baseline, not arbitrary numbers. A 2x increase in auth failures at 2 AM might be normal scaling; at 10 PM, it's an outage.
Comments
Loading comments…