Networks drop. Servers restart. Browser tabs lose focus. When you're building real-time systems with Kafka and WebSockets, these failures aren't edge cases—they're the default state of the world. Reconnection and resilience is what separates a prototype from production, and it's where most teams stumble hardest.
I've watched systems handle spikes flawlessly but crumble when a single region's network hiccups causes 50,000 clients to hammer the server simultaneously during reconnect. This is day 58, and we're going to design for failure deliberately.
Exponential backoff with jitter — Prevents thousands of clients reconnecting at once
When a WebSocket connection drops, the client needs to reconnect. The naive approach is immediate retry. But if 10,000 clients lose connection at the same moment—a network blip, a deploy, a traffic spike killing the server—they'll all try to reconnect immediately. You've just created a thundering herd.
The fix is exponential backoff: wait 1 second, then 2, then 4, then 8, up to some maximum. But exponential backoff alone isn't enough. If all clients use the exact same backoff schedule, they'll still reconnect in synchronized waves. Enter jitter: add randomness to the delay.
Here's a battle-tested implementation:
class WebSocketClient {
private reconnectAttempts = 0;
private maxReconnectAttempts = 12;
private baseDelay = 1000; // 1 second
private maxDelay = 30000; // 30 seconds
disconnect() {
if (this.socket) {
this.socket.close();
this.socket = null;
}
}
private scheduleReconnect() {
if (this.reconnectAttempts >= this.maxReconnectAttempts) {
console.error('Max reconnect attempts reached');
this.onMaxRetriesExceeded();
return;
}
const exponentialDelay = this.baseDelay * Math.pow(2, this.reconnectAttempts);
const cappedDelay = Math.min(exponentialDelay, this.maxDelay);
// Jitter: add ±25% randomness
const jitter = cappedDelay * 0.25 * (Math.random() * 2 - 1);
const finalDelay = cappedDelay + jitter;
console.log(
`Reconnecting in ${Math.round(finalDelay)}ms ` +
`(attempt ${this.reconnectAttempts + 1}/${this.maxReconnectAttempts})`
);
this.reconnectTimeout = setTimeout(() => {
this.reconnectAttempts++;
this.connect();
}, finalDelay);
}
private onConnectionError = () => {
console.warn('WebSocket error, scheduling reconnect');
this.scheduleReconnect();
}
private onConnectionOpen = () => {
console.log('Connected');
this.reconnectAttempts = 0; // Reset on successful connection
this.requestMissedEvents();
}
connect() {
try {
this.socket = new WebSocket(this.url);
this.socket.onopen = this.onConnectionOpen;
this.socket.onerror = this.onConnectionError;
this.socket.onclose = () => this.scheduleReconnect();
this.socket.onmessage = (event) => this.handleMessage(event.data);
} catch (err) {
console.error('Connection failed:', err);
this.scheduleReconnect();
}
}
private requestMissedEvents() {
// Covered in next section
}
private handleMessage(data: string) {
// Covered in idempotency section
}
private onMaxRetriesExceeded() {
// Notify UI or alert user to refresh
window.dispatchEvent(
new CustomEvent('connection-failed', { detail: { permanent: true } })
);
}
}
```
<p>The jitter calculation deserves attention: <code>cappedDelay * 0.25 * (Math.random() * 2 - 1)</code> gives you ±25% variance, spreading reconnect attempts across time. If you have 10,000 clients and a 10-second base delay, they'll naturally spread over a 10-second window instead of slamming the server in one spike.</p>
<h2 id="resume-state-on-reconnect-request-missed-events-since-the-last-known-id">Resume state — On reconnect, request missed events since the last known ID</h2>
<p>Your client dropped for 30 seconds. When it reconnects, it needs to catch up on events it missed. The pattern is: track the last event ID you processed, and on reconnect, ask the server for everything since that point.</p>
<p>This requires coordination between WebSocket and Kafka. When your client receives an event via WebSocket, store its ID. On reconnect, send that ID to the server, which queries Kafka or a state store for missed messages.</p>
<pre><code class="language-typescript">class WebSocketClient {
private lastProcessedEventId: string | null = null;
private socket: WebSocket | null = null;
private handleMessage = (event: MessageEvent) => {
const message = JSON.parse(event.data);
if (message.type === 'event') {
this.processEvent(message);
this.lastProcessedEventId = message.id;
localStorage.setItem('lastEventId', message.id);
}
}
private processEvent(message: any) {
// Your domain logic
console.log('Processing event:', message);
}
private onConnectionOpen = () => {
console.log('Connected');
this.reconnectAttempts = 0;
this.requestMissedEvents();
}
private requestMissedEvents() {
const lastId = this.lastProcessedEventId ||
localStorage.getItem('lastEventId') ||
null;
const payload = {
type: 'resume',
lastEventId: lastId,
clientId: this.clientId,
};
this.socket?.send(JSON.stringify(payload));
}
}
On the server side (NestJS + Kafka), you handle the resume request:
@SubscribeMessage('resume')
async handleResume(
@MessageBody() payload: { lastEventId: string | null; clientId: string },
@ConnectedSocket() client: Socket,
) {
const { lastEventId, clientId } = payload;
try {
let missedEvents = [];
if (lastEventId) {
// Query Kafka for events after lastEventId
missedEvents = await this.kafkaService.getEventsSince(lastEventId);
} else {
// First connection: send recent events or nothing
missedEvents = await this.kafkaService.getRecentEvents(limit: 100);
}
// Send missed events to client
for (const event of missedEvents) {
client.emit('event', event);
}
// Emit a "caught-up" marker
client.emit('caught-up', { at: new Date().toISOString() });
} catch (err) {
client.emit('resume-error', { message: err.message });
}
}
The key insight: you must persist the last event ID, either in localStorage (client) or a database (server). Without it, you can't know where to resume from.
Show connection status — Users should know when data might be stale
A user reads live data, the connection silently drops, and they keep reading stale data thinking it's fresh. That's a bug. Your UI must show connection status.
Emit a status change whenever the connection state changes, and expose it to the UI:
type ConnectionStatus = 'connecting' | 'connected' | 'disconnected' | 'reconnecting';
class WebSocketClient {
private status: ConnectionStatus = 'disconnected';
private statusListeners: Set<(status: ConnectionStatus) => void> = new Set();
private setStatus(status: ConnectionStatus) {
if (this.status === status) return;
this.status = status;
this.statusListeners.forEach(cb => cb(status));
}
onStatusChange(callback: (status: ConnectionStatus) => void) {
this.statusListeners.add(callback);
return () => this.statusListeners.delete(callback);
}
connect() {
this.setStatus('connecting');
this.socket = new WebSocket(this.url);
this.socket.onopen = () => {
this.setStatus('connected');
this.requestMissedEvents();
};
this.socket.onclose = () => {
this.setStatus('disconnected');
this.scheduleReconnect();
};
}
private scheduleReconnect() {
this.setStatus('reconnecting');
setTimeout(() => this.connect(), this.calculateDelay());
}
}
In your React component:
export function DataDisplay() {
const [status, setStatus] = useState('disconnected');
const [data, setData] = useState([]);
useEffect(() => {
const unsubscribe = wsClient.onStatusChange(setStatus);
return unsubscribe;
}, []);
const isStale = status !== 'connected';
return (
<div>
<div>
{status === 'connected' && '✓ Live'}
{status === 'connecting' && '◐ Connecting...'}
{status === 'reconnecting' && '⟳ Reconnecting...'}
{status === 'disconnected' && '✗ Offline'}
</div>
{isStale && (
<div>
Data may be stale. Waiting to reconnect...
</div>
)}
{/* Render your data */}
</div>
);
}
Users see exactly what's happening. No surprises.
Idempotent handlers — Resent messages must not cause duplicate actions
Your client reconnects and receives missed events. But what if some of those events were already processed before the disconnect? Or what if an event arrives twice due to a network retry?
Without idempotency, a charge-to-account event processed twice means charging the user twice. Bad.
The solution: every event needs a unique ID, and your handler must track which IDs you've already processed. Use a set or a database:
class EventProcessor {
private processedEventIds: Set = new Set();
async handleEvent(event: { id: string; type: string; payload: any }) {
// Idempotency check
if (this.processedEventIds.has(event.id)) {
console.log(`Event ${event.id} already processed, skipping`);
return;
}
try {
switch (event.type) {
case 'user.created':
await this.onUserCreated(event.payload);
break;
case 'order.placed':
await this.onOrderPlaced(event.payload);
break;
// ... more handlers
}
this.processedEventIds.add(event.id);
localStorage.setItem(
'processedEvents',
JSON.stringify([...this.processedEventIds])
);
} catch (err) {
console.error(`Failed to process event ${event.id}:`, err);
// Don't mark as processed; retry on next attempt
throw err;
}
}
private async onOrderPlaced(payload: any) {
// This handler is now safe to call multiple times with the same event ID
const { orderId, amount, userId } = payload;
await this.chargeUser(userId, amount);
await this.createShipment(orderId);
}
}
For production, store processed event IDs in a database (Redis with TTL is perfect here) rather than localStorage, so the data persists across app restarts:
async handleEvent(event: { id: string; type: string; payload: any }) {
const alreadyProcessed = await this.redis.get(`event:${event.id}`);
if (alreadyProcessed) {
console.log(`Event ${event.id} already processed`);
return;
}
try {
await this.processEventLogic(event);
// Mark as processed with a 7-day TTL
await this.redis.setex(`event:${event.id}`, 604800, '1');
} catch (err) {
console.error(`Failed to process event ${event.id}:`, err);
throw err;
}
}
Common mistakes
- No jitter in backoff: All clients reconnect in synchronized waves, hammering your server. Use jitter.
- Losing the last event ID: Persisting only in memory means users who refresh the app lose resume state. Use localStorage or a database.
- Assuming the UI knows connection status: Without explicit status signals, your UI can't tell users whether data is live or stale.
- Skipping idempotency checks: "We'll never get duplicate events" is famous last words. Every handler should be idempotent.
- Not testing failures: Throttle your network in dev tools, kill the server, unplug your internet. See how your app behaves. Most issues surface immediately.
Key takeaways
- Exponential backoff with jitter prevents thundering herds and spreads reconnect load naturally across time.
- Track the last processed event ID and request missed events on reconnect to recover gracefully from disconnections.
- Expose connection status to your UI so users know whether they're seeing live or stale data.
- Make every event handler idempotent—check if you've processed an event ID before acting on it.
- Test failures deliberately: disconnect, restart, throttle. Production will do worse.
Comments
Loading comments…