Nginx for WebSockets: a practical Kafka & WebSockets guide

Nginx for WebSockets: The foundation of real-time stability

When you have thousands of concurrent WebSocket connections flowing through your platform, a misconfigured reverse proxy becomes your silent killer. I've watched production incidents unfold because Nginx was stripping upgrade headers or timing out persistent connections at exactly the wrong moment. This is day 62 of the 100-day Kafka & WebSockets series, and we're covering the Nginx configuration that keeps real-time traffic flowing reliably.

The upgrade headers problem

WebSockets begin their life as HTTP requests. A client sends an HTTP/1.1 request with specific headers asking the server to upgrade the protocol to WebSocket. If your reverse proxy doesn't forward these headers correctly, the handshake fails silently and the client falls back to polling—or worse, just times out.

The critical headers are Upgrade and Connection. The client says "I want to upgrade to WebSocket" and the server must echo this back. But Nginx doesn't forward upgrade requests by default. You need to explicitly tell it to pass them through.

Here's the minimal working configuration:

location /socket.io/ {
  proxy_pass http://realtime;
  proxy_http_version 1.1;
  proxy_set_header Upgrade $http_upgrade;
  proxy_set_header Connection "upgrade";
  proxy_read_timeout 3600s;
}

Let me break this down:

• proxy_http_version 1.1: WebSocket requires HTTP/1.1. Nginx defaults to 1.0 for upstream connections, which doesn't support the Upgrade mechanism.
• proxy_set_header Upgrade $http_upgrade: Pass the client's Upgrade header as-is. The variable $http_upgrade contains whatever the client sent (usually "websocket").
• proxy_set_header Connection "upgrade": This tells Nginx to convert the Connection header to "upgrade" for the upstream request. This is essential—Nginx normally uses "Connection: close" by default.
• proxy_read_timeout 3600s: We'll expand on this next, but long-lived WebSocket connections need a long timeout.

I deployed this exact block for a ride-hailing platform's real-time driver tracking system. The moment I added these headers, driver location updates started flowing consistently. Before, about 10% of drivers never received position updates because their WebSocket handshake was failing at the Nginx layer.

Timeouts for long-lived connections

HTTP requests typically complete in milliseconds. WebSocket connections live for minutes, hours, or days. If you don't adjust Nginx's timeout settings, connections will get cut off even though they're actively being used.

The key directive is proxy_read_timeout. This sets how long Nginx will wait for the backend to send data before closing the connection. For WebSocket connections that might go silent for a while (thinking about it, waiting for an event, or just idle), you need a high value.

upstream realtime {
  server backend1:3000;
  server backend2:3000;
  keepalive 64;
}

location /socket.io/ {
  proxy_pass http://realtime;
  proxy_http_version 1.1;
  proxy_set_header Upgrade $http_upgrade;
  proxy_set_header Connection "upgrade";
  
  # Timeout settings for long-lived connections
  proxy_read_timeout 3600s;
  proxy_connect_timeout 7d;
  proxy_send_timeout 3600s;
}

What each timeout does:

• proxy_read_timeout: How long to wait for data from the backend before timing out. Set to 1 hour (3600s) as a baseline. If your application sends heartbeats every 30 seconds, this gives you 2 minutes of buffer.
• proxy_send_timeout: How long to wait before giving up on sending data to the backend. Also 1 hour is reasonable for WebSocket.
• proxy_connect_timeout: How long to wait when initially connecting. This can be shorter, but I often set it high because during peak load, backend startup can take a moment.

The keepalive 64 directive is also crucial. It tells Nginx to maintain up to 64 idle connections to the upstream servers. WebSocket connections are persistent, so reusing these connections reduces overhead significantly.

I learned this lesson painfully on a SaaS platform with real-time notifications. Users would get disconnected after exactly 60 seconds of inactivity. It turned out the default proxy_read_timeout was 60s, and during quiet periods between notifications, the connection timed out. Bumping it to 3600s and implementing application-level heartbeats (every 30 seconds) fixed it completely.

Upstream hashing and connection stickiness

If you have multiple backend servers handling WebSocket connections, you need a strategy to ensure the same client always talks to the same backend. This is called session stickiness or connection affinity.

Why? Because WebSocket is stateful. If client A connects to backend-1 and subscribes to channel X, then suddenly Nginx routes the next request to backend-2, that backend knows nothing about the subscription. You lose state.

Nginx offers several ways to handle this:

IP hash (simple, not perfect)

upstream realtime {
  ip_hash;
  server backend1:3000;
  server backend2:3000;
  server backend3:3000;
}

This hashes the client's IP address and always routes them to the same backend. The problem: if the client is behind a NAT or proxy (common in mobile apps), many clients hash to the same backend, creating load imbalance. Also, if a backend dies, all those clients reconnect and rehash across remaining servers, causing a reconnection storm.

Cookie-based stickiness (better)

upstream realtime {
  server backend1:3000;
  server backend2:3000;
  server backend3:3000;
}

location /socket.io/ {
  proxy_pass http://realtime;
  proxy_http_version 1.1;
  proxy_set_header Upgrade $http_upgrade;
  proxy_set_header Connection "upgrade";
  proxy_read_timeout 3600s;
  
  # Sticky cookie for stickiness
  proxy_cookie_path / "/";
  proxy_cookie_flags ~ secure httponly samesite=lax;
}

On the backend, set a session cookie that identifies which server the client is on. Nginx can then read this cookie and route accordingly (though Nginx open source doesn't have built-in cookie-based routing—you'd use a load balancer like AWS ALB or HAProxy for that). Alternatively, your application can set a custom header that you use for routing.

The distributed state approach (my preference)

Instead of relying on stickiness, share connection state in Redis or a similar system. When client A connects to backend-1 and subscribes to channel X, store this subscription in Redis with the client ID as the key. Now any backend can handle a subsequent request from that client because they can look up the subscription in Redis.

This is more resilient and enables true horizontal scaling. You lose the tight coupling between client and backend server.

In practice, I use stickiness for the initial WebSocket upgrade (keep connection to the same backend that has the open socket), but store subscription and state data in Redis. Best of both worlds.

Worker connections and system limits

Nginx has a pool of worker processes. Each worker can handle a certain number of simultaneous connections. If you don't configure this correctly, you'll hit a wall where new connections are rejected even though your backend has capacity.

user nginx;
worker_processes auto;
worker_connections 4096;
worker_rlimit_nofile 65535;

events {
  worker_connections 4096;
}

http {
  # ... rest of config
}

Key settings:

• worker_processes auto: Nginx will spawn one worker per CPU core. On a 4-core machine, you get 4 workers.
• worker_connections: Each worker can handle this many simultaneous connections. Default is 512, way too low for real-time systems. Set to at least 4096, or higher if you're targeting 10k+ concurrent connections per worker.
• worker_rlimit_nofile: The max number of open file descriptors per worker. Each connection uses one FD, so set this equal to or higher than worker_connections.

Quick math: if you have 4 workers and each can handle 4096 connections, that's 16,384 simultaneous WebSocket connections per Nginx instance. In production, I usually aim for 65,536 file descriptors per worker and bump worker_connections to 32,768 or higher on high-traffic deployments.

I once inherited a real-time trading platform where Nginx was rejecting connections at 1024 simultaneous users. The config had never been touched from the default. Bumping worker_connections to 8192 and worker_rlimit_nofile to 65535 instantly solved the problem—no backend changes needed.

A complete example configuration

Let me put this all together in a realistic setup for a real-time messaging system:

user nginx;
worker_processes auto;
worker_rlimit_nofile 65535;

events {
  worker_connections 8192;
  use epoll; # Linux-specific, high-performance event model
}

http {
  upstream realtime_backend {
    least_conn; # Load balance by active connections
    server backend1:3000 max_fails=3 fail_timeout=30s;
    server backend2:3000 max_fails=3 fail_timeout=30s;
    server backend3:3000 max_fails=3 fail_timeout=30s;
    keepalive 128;
  }

  server {
    listen 80;
    server_name api.example.com;

    location /socket.io/ {
      proxy_pass http://realtime_backend;
      proxy_http_version 1.1;
      
      # Upgrade headers
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection "upgrade";
      
      # Proxying headers
      proxy_set_header Host $host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto $scheme;
      
      # Timeouts for long-lived connections
      proxy_read_timeout 3600s;
      proxy_send_timeout 3600s;
      proxy_connect_timeout 7d;
      
      # Buffering
      proxy_buffering off;
    }

    # Regular HTTP API endpoints
    location /api/ {
      proxy_pass http://realtime_backend;
      proxy_http_version 1.1;
      proxy_set_header Host $host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_read_timeout 30s;
    }
  }
}

Note the proxy_buffering off directive for WebSocket. Buffering is for HTTP requests and responses, not persistent connections. Turning it off ensures immediate delivery of frames.

Common mistakes

Missing proxy_http_version 1.1: This alone will break WebSocket. The upgrade mechanism doesn't exist in HTTP/1.0.

Setting Connection: close: Some configs explicitly set this header. Don't. It closes the connection after every response, defeating the purpose of WebSocket.

Forgetting to adjust file descriptor limits: You can set worker_connections to 100,000, but if the OS limit is 1024, you'll still cap out early. Always check ulimit -n on your Nginx server and ensure it's high enough.

Using buffering with WebSocket: Buffering introduces latency and breaks frame delivery. proxy_buffering off is essential.

Ignoring heartbeats: If your application doesn't send heartbeat frames periodically, idle connections may be closed by firewalls or intermediate proxies. Aim for a heartbeat every 20–30 seconds.

No monitoring: Set up alerts on Nginx's connection counts, active connections, and worker processes. If you're consistently near the limit, you need to scale horizontally.

Key takeaways

• Always set proxy_http_version 1.1 and forward the Upgrade and Connection headers—these are non-negotiable for WebSocket handshakes.
• Use proxy_read_timeout and proxy_send_timeout values of 3600s or higher for WebSocket connections, since they're long-lived and may go silent for extended periods.
• Plan for stickiness: either use IP hash for simplicity, rely on cookies, or better yet, distribute state across Redis so any backend can serve any client.
• Configure worker_connections and OS file descriptor limits based on your peak concurrent connection count. A 4-core server with 8192 connections per worker can handle ~32k WebSocket clients.
• Disable buffering (proxy_buffering off) for WebSocket traffic to ensure real-time frame delivery without artificial latency.
• Use least_conn load balancing for WebSocket upstreams—it balances based on active connections, not round-robin.
• Add health checks and failure detection (max_fails, fail_timeout) so a dead backend doesn't break the entire load balancer.