Test reconnect storms for Kafka & WebSockets developers
When a production deployment goes wrong or infrastructure fails, thousands of clients don't reconnect one by one—they all try at once. A reconnect storm can overwhelm your auth services, exhaust your Kafka broker connections, and cascade into a complete system outage. In this article, we'll walk through how to simulate, detect, and survive reconnect storms so your team can deploy with confidence.
Understanding reconnect storms at scale
A reconnect storm happens when a large cohort of WebSocket clients simultaneously lose their connection and attempt to re-establish it. This is the real-world scenario: imagine 50,000 mobile app users whose server connection drops due to a rolling deploy, network partition, or a misconfigured load balancer. Without proper safeguards, those 50,000 clients will hammer your authentication endpoint, exhaust your connection pool, and trigger cascading failures across downstream services.
The danger isn't theoretical. We've seen teams lose 30 minutes of production time because a single deploy didn't account for connection backpressure. The fix starts with testing before you need it.
Simulate a server restart under load
You can't fix what you haven't measured. The first step is to build a test harness that simulates realistic client behavior, including reconnection attempts after a server restart.
Start with a simple load test that opens WebSocket connections to your server, then forcibly restarts the server and measures what happens next:
typescript
import WebSocket from 'ws';
interface ClientMetrics {
id: string;
connectedAt: number | null;
reconnectAttempts: number;
lastError: string | null;
state: 'connecting' | 'connected' | 'disconnected' | 'failed';
}
class ReconnectStormSimulator {
private clients: Map<string, ClientMetrics> = new Map();
private metrics = {
totalConnections: 0,
successfulReconnects: 0,
failedReconnects: 0,
peakConcurrentReconnects: 0,
authErrors: 0,
timeToRecover: 0,
};
async simulateStorm(clientCount: number, serverUrl: string) {
console.log(`Starting storm simulation with ${clientCount} clients`);
// Phase 1: Establish baseline connections
const clients = await this.establishConnections(clientCount, serverUrl);
console.log(`[Phase 1] Established ${clients.length} baseline connections`);
// Phase 2: Wait for stability
await new Promise(resolve => setTimeout(resolve, 2000));
// Phase 3: Simulate server restart
console.log(`[Phase 2] Simulating server restart...`);
const stormStart = Date.now();
await this.triggerReconnectionWave(clients);
// Phase 4: Monitor recovery
const recovered = await this.monitorRecovery(clients);
const timeToRecover = Date.now() - stormStart;
this.metrics.timeToRecover = timeToRecover;
this.printMetrics();
return recovered;
}
private async establishConnections(
count: number,
serverUrl: string
): Promise<ClientMetrics[]> {
const clients: ClientMetrics[] = [];
for (let i = 0; i < count; i++) {
const client: ClientMetrics = {
id: `client-${i}`,
connectedAt: null,
reconnectAttempts: 0,
lastError: null,
state: 'connecting',
};
this.clients.set(client.id, client);
clients.push(client);
// Stagger connection attempts to avoid initial thundering herd
if (i % 100 === 0) {
await new Promise(resolve => setTimeout(resolve, 10));
}
}
return clients;
}
private async triggerReconnectionWave(clients: ClientMetrics[]) {
// Simulate all clients losing connection simultaneously
for (const client of clients) {
client.state = 'disconnected';
client.connectedAt = null;
client.reconnectAttempts = 0;
}
// All clients attempt to reconnect at roughly the same time
const reconnectPromises = clients.map(client =>
this.attemptReconnect(client)
);
await Promise.allSettled(reconnectPromises);
}
private async attemptReconnect(client: ClientMetrics) {
const maxRetries = 5;
let backoffMs = 100;
while (client.reconnectAttempts < maxRetries) {
try {
client.state = 'connecting';
client.reconnectAttempts++;
// Wait with exponential backoff
await new Promise(resolve =>
setTimeout(resolve, backoffMs + Math.random() * backoffMs * 0.1)
);
// Simulate reconnection attempt
// In real tests, this would be an actual WebSocket.connect()
client.state = 'connected';
client.connectedAt = Date.now();
this.metrics.successfulReconnects++;
return;
} catch (error) {
client.lastError = String(error);
backoffMs = Math.min(backoffMs * 2, 5000);
this.metrics.failedReconnects++;
}
}
client.state = 'failed';
}
private async monitorRecovery(clients: ClientMetrics[]): Promise<number> {
const maxWaitMs = 60000; // 60 second timeout
const startTime = Date.now();
while (Date.now() - startTime < maxWaitMs) {
const connected = clients.filter(c => c.state === 'connected').length;
const failed = clients.filter(c => c.state === 'failed').length;
if (connected + failed === clients.length) {
return connected;
}
await new Promise(resolve => setTimeout(resolve, 500));
}
return clients.filter(c => c.state === 'connected').length;
}
private printMetrics() {
console.log('\n=== Reconnect Storm Test Results ===');
console.log(`Total clients: ${this.clients.size}`);
console.log(`Successful reconnects: ${this.metrics.successfulReconnects}`);
console.log(`Failed reconnects: ${this.metrics.failedReconnects}`);
console.log(`Time to recover: ${this.metrics.timeToRecover}ms`);
console.log(`Auth errors: ${this.metrics.authErrors}`);
}
}
// Usage
const simulator = new ReconnectStormSimulator();
await simulator.simulateStorm(50000, 'ws://localhost:8080');
This test establishes a baseline of connections, then forces all clients to reconnect at once. Run it against your staging environment and watch for bottlenecks. Pay attention to:
- Auth service response time: Does latency spike during reconnection?
- Database connection pool saturation: Are you running out of connections?
- Kafka producer failures: Are messages being dropped due to producer backpressure?
- Memory growth: Does your server leak connections or buffers?
Verify backoff and jitter work
Without proper exponential backoff and jitter, your reconnection storm never actually recovers—it just keeps hammering your services. Jitter ensures that clients don't all retry at exactly the same moment.
Here's how to verify that backoff and jitter reduce peak load:
typescript
class BackoffValidator {
async testBackoffDistribution() {
const attemptTimestamps: number[] = [];
const clientCount = 1000;
// Simulate 1000 clients all attempting to reconnect
const promises = Array.from({ length: clientCount }).map(async (_, i) => {
const baseBackoff = 100 + (i % 5) * 100; // Initial backoff: 100-500ms
const jitter = Math.random() * baseBackoff * 0.1; // 10% jitter
const delay = baseBackoff + jitter;
const timestamp = Date.now();
await new Promise(resolve => setTimeout(resolve, delay));
attemptTimestamps.push(Date.now() - timestamp);
});
await Promise.all(promises);
// Analyze distribution
const bucketed = this.bucketByTime(attemptTimestamps, 100);
console.log('Reconnection attempts per 100ms bucket:');
bucketed.forEach((count, bucket) => {
console.log(`${bucket * 100}-${(bucket + 1) * 100}ms: ${count} attempts`);
});
const peakLoad = Math.max(...bucketed.values());
const evenLoad = clientCount / bucketed.size;
console.log(`Peak load: ${peakLoad} (ideal: ~${evenLoad})`);
console.log(`Load distribution ratio: ${(peakLoad / evenLoad).toFixed(2)}x`);
}
private bucketByTime(
timestamps: number[],
bucketSize: number
): Map<number, number> {
const buckets = new Map<number, number>();
timestamps.forEach(ts => {
const bucket = Math.floor(ts / bucketSize);
buckets.set(bucket, (buckets.get(bucket) || 0) + 1);
});
return buckets;
}
}
const validator = new BackoffValidator();
await validator.testBackoffDistribution();
A good jitter implementation spreads reconnection attempts evenly over time. If your peak load is only 1.5x the average, your backoff strategy is working. If it's 10x, clients are still synchronized and you haven't solved the problem.
Protect auth services from spikes
Your auth service is the choke point. Every reconnecting client needs to authenticate, and a spike in requests will queue up and timeout if you're not prepared.
Implement rate limiting and circuit breakers at the auth layer:
typescript
class AuthServiceProtection {
private rateLimiter = {
tokensPerSecond: 500,
tokens: 500,
lastRefillTime: Date.now(),
};
private circuitBreaker = {
state: 'closed' as 'closed' | 'open' | 'half-open',
failureCount: 0,
failureThreshold: 10,
successThreshold: 5,
lastFailureTime: 0,
resetTimeoutMs: 30000,
};
async authenticateWithProtection(
userId: string,
token: string
): Promise<{ success: boolean; reason?: string }> {
// Check circuit breaker
if (this.circuitBreaker.state === 'open') {
if (Date.now() - this.circuitBreaker.lastFailureTime >
this.circuitBreaker.resetTimeoutMs) {
this.circuitBreaker.state = 'half-open';
} else {
return {
success: false,
reason: 'Auth service temporarily unavailable (circuit open)'
};
}
}
// Check rate limit
if (!this.allowRequest()) {
return {
success: false,
reason: 'Rate limit exceeded'
};
}
try {
const result = await this.callAuthService(userId, token);
if (result.success) {
this.onAuthSuccess();
} else {
this.onAuthFailure();
}
return result;
} catch (error) {
this.onAuthFailure();
return {
success: false,
reason: 'Auth service error'
};
}
}
private allowRequest(): boolean {
const now = Date.now();
const timeSinceRefill = now - this.rateLimiter.lastRefillTime;
const tokensToAdd = (timeSinceRefill / 1000) *
this.rateLimiter.tokensPerSecond;
this.rateLimiter.tokens = Math.min(
this.rateLimiter.tokensPerSecond,
this.rateLimiter.tokens + tokensToAdd
);
this.rateLimiter.lastRefillTime = now;
if (this.rateLimiter.tokens >= 1) {
this.rateLimiter.tokens -= 1;
return true;
}
return false;
}
private onAuthSuccess() {
if (this.circuitBreaker.state === 'half-open') {
this.circuitBreaker.successThreshold--;
if (this.circuitBreaker.successThreshold <= 0) {
this.circuitBreaker.state = 'closed';
this.circuitBreaker.failureCount = 0;
}
} else {
this.circuitBreaker.failureCount = 0;
}
}
private onAuthFailure() {
this.circuitBreaker.failureCount++;
this.circuitBreaker.lastFailureTime = Date.now();
if (this.circuitBreaker.failureCount >=
this.circuitBreaker.failureThreshold) {
this.circuitBreaker.state = 'open';
}
}
private async callAuthService(
userId: string,
token: string
): Promise<{ success: boolean }> {
// Call your actual auth service
return { success: true };
}
}
This approach prevents your auth service from being overwhelmed. Requests that exceed your capacity are rejected cleanly, rather than timing out and creating more retry pressure.
Stagger deploys across instances
One of the most common triggers for a reconnect storm is a rolling deploy. If you restart all your WebSocket servers at once, every connected client will reconnect simultaneously. Instead, stagger your deploys so only a fraction of clients disconnect at a time.
Here's a deployment coordinator that spreads server restarts over time:
typescript
interface DeploymentConfig {
totalInstances: number;
batchSize: number;
delayBetweenBatchesMs: number;
healthCheckIntervalMs: number;
maxHealthCheckRetries: number;
}
class StaggedDeploymentCoordinator {
private config: DeploymentConfig;
constructor(config: DeploymentConfig) {
this.config = config;
}
async orchestrateRollingDeploy(instances: string[]): Promise<void> {
const batches = this.createBatches(instances);
for (let i = 0; i < batches.length; i++) {
const batch = batches[i];
console.log(
`[Batch ${i + 1}/${batches.length}] ` +
`Deploying to ${batch.join(', ')}`
);
// Give clients time to reconnect before moving to next batch
await this.deployBatch(batch);
if (i < batches.length - 1) {
console.log(
`Waiting ${this.config.delayBetweenBatchesMs}ms ` +
`before next batch`
);
await new Promise(resolve =>
setTimeout(resolve, this.config.delayBetweenBatchesMs)
);
}
}
}
private createBatches(instances: string[]): string[][] {
const batches: string[][] = [];
for (let i = 0; i < instances.length; i += this.config.batchSize) {
batches.push(instances.slice(i, i + this.config.batchSize));
}
return batches;
}
private async deployBatch(instances: string[]): Promise<void> {
const deployPromises = instances.map(instance =>
this.deployToInstance(instance)
);
const results = await Promise.allSettled(deployPromises);
const failures = results.filter(r => r.status === 'rejected');
if (failures.length > 0) {
console.error(`${failures.length} deployments failed in batch`);
}
}
private async deployToInstance(instance: string): Promise<void> {
for (let attempt = 0; attempt < this.config.maxHealthCheckRetries;
attempt++) {
try {
// Drain connections gracefully before restart
await this.drainConnections(instance);
// Restart the instance
await this.restartInstance(instance);
// Verify health
await this.waitForHealthy(instance);
return;
} catch (error) {
console.error(
`Attempt ${attempt + 1} failed for ${instance}:`,
error
);
if (attempt === this.config.maxHealthCheckRetries - 1) {
throw error;
}
await new Promise(resolve => setTimeout(resolve, 2000));
}
}
}
private async drainConnections(instance: string): Promise<void> {
console.log(`Draining connections on ${instance}`);
// Send drain signal to instance
// Existing connections close gracefully
// New connections are rejected
await new Promise(resolve => setTimeout(resolve, 1000));
}
private async restartInstance(instance: string): Promise<void> {
console.log(`Restarting ${instance}`);
await new Promise(resolve => setTimeout(resolve, 2000));
}
private async waitForHealthy(instance: string): Promise<void> {
console.log(`Waiting for ${instance} to become healthy`);
await new Promise(resolve => setTimeout(resolve, 3000));
}
}
// Usage
const coordinator = new StaggedDeploymentCoordinator({
totalInstances: 20,
batchSize: 3,
delayBetweenBatchesMs: 10000,
healthCheckIntervalMs: 1000,
maxHealthCheckRetries: 5,
});
await coordinator.orchestrateRollingDeploy([
'ws-1', 'ws-2', 'ws-3', /* ... more instances ... */
]);
By deploying in batches of 2-5 instances with delays between batches, you reduce the peak reconnection load from "all 50,000 clients" to "10% of clients at a time." This makes a dramatic difference in whether your system recovers smoothly.
Common mistakes
- Testing with too few clients: A load test with 1,000 clients won't show you what 50,000 look like. Scale your simulations to match your actual user base or larger.
- Ignoring Kafka producer backpressure: When clients reconnect, they often re-subscribe to topics. Kafka brokers can reject batches if producers push too hard. Test end-to-end with Kafka, not just WebSocket connections.
- Jitter without exponential backoff: Jitter alone spreads the storm; exponential backoff prevents repeated waves of retries. Use both.
- No visibility into what's happening: Add structured logging and metrics so you can see when a reconnection storm is starting. Track concurrent connections, auth latency, and Kafka lag in real time.
- Failing to document deploy procedures: Your team won't stagger deploys if they don't know why. Document the risks and the correct procedure so on-call engineers follow it under pressure.
Key takeaways
- Reconnect storms are predictable and testable. Simulate them in staging before they hit production.
- Exponential backoff with jitter is essential. Verify your implementation actually spreads load evenly, not just in theory.
- Protect your auth services with rate limiting and circuit breakers. They're the first thing to fail under spike load.
- Stagger rolling deploys so only a fraction of clients reconnect at once. Batch sizes of 2–5 instances are typical; adjust based on your load test results.
- Add comprehensive metrics for connection storms: concurrent reconnects, auth latency, producer failures, and recovery time. You need visibility to act quickly.
- Document your deploy procedure and reconnection safeguards so the team follows them consistently.
Comments
Loading comments…