Heartbeats keep connections honest
A connection can look perfectly healthy until the moment you actually need to send data through it. Dead sockets, half-open TCP connections, and proxies that silently drop idle traffic create a false sense of security—your application thinks it's connected when it's already broken. Heartbeats keep connections honest by forcing periodic validation, catching silent failures before they cascade into user-facing outages.
The problem: dead connections masquerading as alive
In distributed systems, a TCP connection can enter a zombie state. The local socket remains open, the application layer has no idea anything is wrong, but packets sent into the void simply vanish. This happens because TCP doesn't validate that the remote end is still there—it only validates that packets arrive. If no packets are sent, no validation occurs.
I learned this the hard way during a ride-hailing project. Drivers maintained long-lived WebSocket connections to receive ride requests. Some connections would silently die—a network partition, a proxy timeout, a NAT table entry expiring—but the driver app wouldn't know. Users requested rides, the system thought the driver was connected, and rides timed out after 30 seconds. Without heartbeats, we had no way to detect these failures in real time.
The fix is straightforward: if we don't hear from a client, we force them to prove they're alive. This is exactly what heartbeats do.
Ping at a regular interval
A heartbeat is simply a small message sent at a regular interval—typically 10 to 30 seconds for WebSocket connections, though this varies by use case. The server sends a PING frame, and the client responds with a PONG. If the connection is alive and the client is responsive, the PONG comes back within milliseconds. If it doesn't, something is broken.
Here's a practical implementation using NestJS with WebSocket support:
import {
WebSocketGateway,
WebSocketServer,
SubscribeMessage,
OnGatewayConnection,
OnGatewayDisconnect,
} from '@nestjs/websockets';
import { Server, Socket } from 'socket.io';
@WebSocketGateway({
cors: { origin: '*' },
pingInterval: 10000, // Ping every 10 seconds
pingTimeout: 5000, // Wait 5 seconds for pong
})
export class RideGateway implements OnGatewayConnection, OnGatewayDisconnect {
@WebSocketServer()
server: Server;
private heartbeatIntervals = new Map();
handleConnection(socket: Socket) {
console.log(`Driver connected: ${socket.id}`);
// Socket.io handles ping/pong automatically with pingInterval
// But we can add custom logic if needed
socket.on('pong', () => {
console.log(`Pong received from ${socket.id}`);
});
}
handleDisconnect(socket: Socket) {
console.log(`Driver disconnected: ${socket.id}`);
const interval = this.heartbeatIntervals.get(socket.id);
if (interval) {
clearInterval(interval);
this.heartbeatIntervals.delete(socket.id);
}
}
@SubscribeMessage('ride-accepted')
handleRideAccepted(socket: Socket, data: any) {
console.log(`Ride accepted by ${socket.id}`);
return { status: 'ok' };
}
}
Socket.io handles heartbeats automatically with the pingInterval and pingTimeout options. The server sends a PING every 10 seconds, and if no PONG arrives within 5 seconds, the connection is considered dead. This is production-ready out of the box.
For lower-level control, you might implement heartbeats over raw WebSockets or with Kafka consumer groups. The principle remains the same: send a lightweight message at regular intervals and expect a response.
Close connections that miss pongs
A heartbeat is only useful if you act on the response. If a client misses a PONG window, you must close the connection immediately. Keeping dead connections open wastes resources and delays the reconnection cycle that would restore service to the user.
With Socket.io, this happens automatically when pingTimeout is exceeded. The server closes the socket, triggers handleDisconnect, and the client reconnects. But if you're implementing heartbeats manually, you need explicit cleanup:
import WebSocket from 'ws';
class CustomHeartbeatManager {
private socket: WebSocket;
private heartbeatTimeout: NodeJS.Timeout;
private lastPong: number;
readonly HEARTBEAT_INTERVAL = 10000; // 10 seconds
readonly PONG_TIMEOUT = 5000; // 5 seconds to respond
constructor(socket: WebSocket) {
this.socket = socket;
this.lastPong = Date.now();
this.setupHeartbeat();
this.socket.on('pong', () => this.handlePong());
this.socket.on('close', () => this.cleanup());
}
private setupHeartbeat() {
setInterval(() => {
if (this.socket.readyState === WebSocket.OPEN) {
console.log('Sending ping');
this.socket.ping();
// Set a timeout for the pong response
this.heartbeatTimeout = setTimeout(() => {
console.error('Pong timeout - closing connection');
this.socket.close(1000, 'Heartbeat timeout');
}, this.PONG_TIMEOUT);
}
}, this.HEARTBEAT_INTERVAL);
}
private handlePong() {
console.log('Pong received');
if (this.heartbeatTimeout) {
clearTimeout(this.heartbeatTimeout);
}
this.lastPong = Date.now();
}
private cleanup() {
if (this.heartbeatTimeout) {
clearTimeout(this.heartbeatTimeout);
}
}
}
The key insight: silence is not golden. If you don't hear a PONG, assume the worst and close the connection. The client will reconnect, and you'll discover whether it's truly dead or just slow to respond.
Keeps proxies from timing out idle sockets
Network proxies, load balancers, and firewalls often drop idle TCP connections after a fixed period (commonly 5–15 minutes). These devices don't know if your application is intentionally keeping a connection open for low-frequency messages or if it's truly dead. To them, silence looks suspicious.
A heartbeat solves this by generating traffic at predictable intervals. Even if no application data flows for hours, the ping/pong conversation proves the connection is in use. Proxies see regular activity and leave the socket alone.
In the trading platform I built, clients maintained long-lived connections for market data updates. Market could be quiet for 10+ minutes overnight, but clients stayed subscribed. Without heartbeats, proxies would kill these connections, and clients would lose their subscription without realizing it. With heartbeats every 15 seconds, the proxies had no reason to drop the socket.
This is especially critical in mobile and remote office environments where NAT devices and corporate proxies are common. A heartbeat is cheap insurance against proxy-induced disconnections.
Detects half-open TCP connections
A half-open connection is the most insidious failure state. One side of the TCP connection is open, the other side is closed. The side with the open socket has no way to know the remote end is gone unless it tries to send data. If it never sends data, it never discovers the failure.
Picture this scenario: a Kafka consumer subscribes to a topic and receives one message. It processes the message, then waits for the next one. If no new messages arrive, the connection goes silent. If the broker crashes or the network partition occurs during this quiet period, the consumer doesn't know—until it tries to send a heartbeat.
Heartbeats force the issue. When the server sends a PING and gets no PONG, it has empirical evidence that the connection is broken. Without heartbeats, the consumer would sit in a zombie state indefinitely.
Here's how Kafka clients handle this with automatic heartbeats:
import { Kafka } from 'kafkajs';
const kafka = new Kafka({
clientId: 'consumer-app',
brokers: ['broker1:9092', 'broker2:9092'],
retry: {
initialRetryTime: 100,
retries: 8,
multiplier: 2,
randomizationFactor: 0.2,
},
requestTimeout: 30000,
connectionTimeout: 10000,
// Heartbeat configuration
ssl: false,
sasl: undefined,
});
const consumer = kafka.consumer({
groupId: 'ride-updates',
heartbeatInterval: 6000, // Send heartbeat every 6 seconds
sessionTimeout: 30000, // Broker expects heartbeat within 30 seconds
});
await consumer.connect();
await consumer.subscribe({ topic: 'ride-status', fromBeginning: false });
await consumer.run({
eachMessage: async ({ topic, partition, message }) => {
console.log('Processing message:', message.value.toString());
},
});
The Kafka consumer sends heartbeats to the broker every 6 seconds. If it misses 5 consecutive heartbeats (30-second session timeout), the broker evicts it from the consumer group and rebalances. The client discovers the failure and reconnects. Without this, a crashed consumer would clog the group for minutes or hours.
Common mistakes
Setting heartbeat intervals too long. A 5-minute heartbeat won't catch proxy timeouts at the 10-minute mark. Keep intervals short—10 to 30 seconds for WebSockets, 3 to 10 seconds for Kafka. Shorter intervals catch failures faster and consume minimal bandwidth.
Forgetting to actually close the connection. Some teams log a timeout but leave the socket open, hoping it will recover. It won't. Close it and let the client reconnect. Graceful recovery beats zombie connections every time.
Using heartbeats as a substitute for proper error handling. Heartbeats detect lost connections, but they don't fix application errors. A timeout on a database query needs its own timeout and retry logic. Heartbeats are orthogonal to application-level error handling.
Ignoring platform defaults. Socket.io, gRPC, and Kafka all have built-in heartbeat mechanisms. Override them only if you have a specific reason. The defaults are tuned by teams with billions of connections.
Key takeaways
- Heartbeats keep connections honest by forcing periodic validation. A connection that looks alive may be dead.
- Implement heartbeats with a short, regular interval (10–30 seconds for WebSockets, 6 seconds for Kafka) and a tight timeout window (3–5 seconds).
- Close connections that miss pongs immediately. Silence means failure; restart the connection cycle.
- Heartbeats prevent proxies and firewalls from timing out idle connections that should stay open for low-frequency traffic.
- Half-open TCP connections are invisible without heartbeats. A periodic PING/PONG forces them into the open.
- Use platform defaults (Socket.io's
pingInterval, Kafka'sheartbeatInterval) unless you have a reason not to. - Heartbeats are not a substitute for application-level timeouts and retries. Use both.
Comments
Loading comments…