Connection limits and file descriptors, explained simply

Connection limits and file descriptors

Every socket is an open file. When you're building real-time systems with Kafka and WebSockets, you'll hit resource exhaustion long before your code fails. I've seen production outages caused by ulimit caps that were never questioned, worker connection pools sized for yesterday's traffic, and memory leaks that only appeared under realistic load.

This isn't theoretical—it's the difference between a system that scales gracefully and one that crashes at 3 AM when concurrent users spike. Let me walk you through what I've learned the hard way over the past five years building e-commerce platforms, ride-hailing systems, and SaaS applications that needed to handle thousands of simultaneous connections.

Why file descriptors matter for socket servers

Unix treats everything as a file. A network socket, a database connection, an open log file—they're all file descriptors. Your operating system has a hard limit on how many a single process can hold at once. This is controlled by ulimit -n.

When you launch a Node.js or .NET process without raising this limit, you're typically capped at 1,024 descriptors on Linux. That sounds like a lot until you realize a WebSocket connection uses at least one descriptor. Add in database connections, Redis connections, and file I/O, and you'll exhaust your budget fast.

I ran into this on a travel booking platform I built with NestJS. We were handling 5,000 concurrent WebSocket connections but crashing at around 800. The error was cryptic: "EMFILE: too many open files." The fix was simple but crucial:

// Before: ulimit -n 1024 (default)
// After: ulimit -n 65536

// In your NestJS bootstrap, verify it's set:
console.log('Max file descriptors:', require('os').constants.UV_MAXSOCKETS);

// Better: explicitly set in Node
const os = require('os');
const maxConnections = Math.min(65536, os.constants.UV_MAXSOCKETS);
server.maxConnections = maxConnections;

On production servers, I always set ulimit during container startup or system configuration:

// Docker example - set in Dockerfile or docker-compose
// RUN echo "* soft nofile 65536" >> /etc/security/limits.conf
// RUN echo "* hard nofile 65536" >> /etc/security/limits.conf

// Kubernetes - set in Pod spec
// securityContext:
//   fsGroup: 1000
// containers:
//   - name: app
//     securityContext:
//       capabilities:
//         add:
//           - SYS_RESOURCE

The rule I follow: set ulimit to at least 2x your expected concurrent connections. If you're targeting 10,000 WebSocket clients, set it to 20,000 or higher. It's free and prevents a whole category of surprises.

Tuning worker connections in proxies

Your WebSocket server isn't the only place connections matter. When you're running load balancers (Nginx, HAProxy) or API gateways in front of your Kafka consumers, they also need tuning.

Nginx, for example, has a worker_connections directive that defaults to 512. That's per worker process. If you're running with 4 workers, your total capacity is 2,048 connections. Beyond that, incoming requests queue or get rejected.

For a SaaS platform I built with Next.js and Kafka, we had Nginx proxying WebSocket upgrades to a NestJS backend. Under load testing, browsers would fail to connect when we hit around 2,000 concurrent users—despite the backend having capacity.

The Nginx configuration needed adjustment:

// nginx.conf before (inadequate)
events {
  worker_connections 512;
}

// nginx.conf after (production-ready)
events {
  worker_connections 10000;  // Per worker
  use epoll;                 // Linux only, more efficient than default select
  multi_accept on;
}

// Also tune buffer sizes
http {
  client_body_buffer_size 10M;
  client_max_body_size 100M;
  
  upstream backend {
    keepalive 64;  // Reuse backend connections
    server app:3000 max_fails=3 fail_timeout=30s;
  }
  
  server {
    location / {
      proxy_pass http://backend;
      proxy_http_version 1.1;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection "upgrade";
      proxy_set_header Connection "";
      proxy_buffering off;
    }
  }
}

I also tune HAProxy similarly when it's the choice. The key is the maxconn directive:

// haproxy.cfg
global
  maxconn 200000
  
defaults
  timeout connect 5s
  timeout client 60s
  timeout server 60s

listen websocket_backend
  bind *:8080
  maxconn 100000
  balance roundrobin
  server app1 app1:3000 maxconn 10000
  server app2 app2:3000 maxconn 10000

The lesson: proxy limits are invisible until you hit them. I always load test proxies separately from application servers to catch these misconfigurations early.

Monitor memory per connection

A connection isn't free. Each WebSocket holds buffers, state, timers, and event listeners. Memory usage is a multiplier of connection count.

I've debugged platforms where 10,000 connections consumed 16 GB of RAM, and others where the same connection count used 4 GB. The difference: memory leaks, inefficient message buffering, and unbounded caches tied to connection lifetime.

For one ride-hailing system I built with WebSockets, we tracked memory per connection obsessively:

// NestJS WebSocket gateway with memory monitoring
import { WebSocketGateway, WebSocketServer, OnGatewayConnection } from '@nestjs/websockets';

@WebSocketGateway({ namespace: 'ride' })
export class RideGateway implements OnGatewayConnection {
  @WebSocketServer() server: Server;
  private connections = new Map();

  async handleConnection(client: Socket) {
    const before = process.memoryUsage().heapUsed;
    
    this.connections.set(client.id, {
      joinedAt: Date.now(),
      subscriptions: new Set(),
      bufferSize: 0,
    });
    
    const after = process.memoryUsage().heapUsed;
    const delta = (after - before) / 1024 / 1024;
    
    console.log(`Connection: ${this.connections.size} total, +${delta.toFixed(2)}MB`);
    
    // Alert if single connection > 5MB (memory leak signal)
    if (delta > 5) {
      console.warn(`Suspicious memory spike for ${client.id}: ${delta.toFixed(2)}MB`);
    }
  }

  handleDisconnect(client: Socket) {
    this.connections.delete(client.id);
    console.log(`Disconnected: ${this.connections.size} remaining`);
  }
}

In production, I pair this with container monitoring. For a Docker setup, I track memory per pod:

// Prometheus metrics export
import { Registry, Gauge, Counter } from 'prom-client';

const connectionGauge = new Gauge({
  name: 'websocket_connections_total',
  help: 'Total WebSocket connections',
  registers: [registry],
});

const memoryPerConnectionGauge = new Gauge({
  name: 'memory_per_connection_bytes',
  help: 'Heap memory consumed per connection',
  registers: [registry],
});

setInterval(() => {
  const mem = process.memoryUsage();
  const conns = this.connections.size;
  const perConn = conns > 0 ? mem.heapUsed / conns : 0;
  
  connectionGauge.set(conns);
  memoryPerConnectionGauge.set(perConn);
}, 5000);

The pattern I use: if memory per connection exceeds 1 MB, investigate. Most well-written WebSocket handlers use 50–200 KB per connection. Anything higher suggests queued messages, retained references, or listener leaks.

Load test before you need it

Theory fails under pressure. I always load test connection limits, file descriptor exhaustion, and memory scaling before deploying to production.

For Kafka consumers feeding WebSocket broadcasts, I simulate realistic scenarios with tools like Artillery or k6:

// Artillery config: load-test.yml
config:
  target: "http://localhost:3000"
  phases:
    - duration: 60
      arrivalRate: 100  # 100 new connections/sec
      ramp: 50          # Gradually increase by 50/sec
    - duration: 300
      arrivalRate: 200  # Hold at 200/sec for 5 minutes
    - duration: 60
      arrivalRate: 0    # Ramp down

scenarios:
  - name: "WebSocket Load"
    flow:
      - connect:
          target: "ws://localhost:3000/ws"
          beforeConnect: "setup"
      - think: 2
      - send: '{"action":"subscribe","topic":"rides"}'
      - think: 5
      - send: '{"action":"update","rideId":"123"}'

I run this against a staging replica and watch for:

• Connection rejection rates (should be 0%)
• Response latency degradation (p99 latency shouldn't spike)
• Memory growth trajectory (should stabilize, not grow unbounded)
• CPU usage and worker saturation
• Database connection pool exhaustion

For a trade execution system I built, load testing revealed that after 15,000 concurrent connections, our database's connection pool was exhausted. We weren't running out of file descriptors or memory—we were blocked at the database. Catching this during staging prevented a production disaster.

I document findings in a simple spreadsheet:

This test told me: we can safely target 10,000 connections on the current infrastructure. To go higher, we need to pool database connections more aggressively (using PgBouncer or similar) or add more database replicas for read queries.

Common mistakes

• Ignoring ulimit on production. Setting it locally and assuming it carries to containers is a classic error. Verify ulimit -n in your running process every time.
• Confusing worker processes with connections. In Nginx, HAProxy, or Node.js cluster mode, total capacity is (worker_connections or maxconn) × (number of workers). I've seen teams assume a single global cap and then wonder why they lose connections.
• Not accounting for non-socket file descriptors. Database connections, file handles, and pipes all count. A 10,000-descriptor limit with 5,000 WebSockets, 2,000 database connections, and logging can be tight.
• Load testing with unrealistic patterns. Constant message rates aren't real. Real traffic is bursty. I simulate traffic spikes, connection storms, and slow clients disconnecting.
• Assuming memory is stable. Monitor over hours, not minutes. Memory leaks often appear after thousands of connection cycles.

Key takeaways

• Raise ulimit -n to at least 2× your expected concurrent connection count. Set it in Docker, Kubernetes, or systemd—don't rely on defaults.
• Tune proxy worker connections and keepalive settings. An inadequate proxy is invisible until load testing reveals it.
• Monitor memory per connection in production. Aim for 50–200 KB per WebSocket; anything higher is a red flag.
• Load test connection limits, memory scaling, and database pool exhaustion before scaling up. Use realistic traffic patterns (spikes, slow clients, connection churn).
• Document your findings. Know the exact connection ceiling for your current infrastructure and what changes will increase it.