Load balancing, explained simply: lessons from production

A single server can only handle so much traffic before it starts dropping requests, timing out, or crashing altogether. What happens when your application grows? You could buy a bigger machine, but that only delays the problem. The real solution is to spread the load across multiple servers, and that's where load balancing comes in. Load balancing, explained simply, is the practice of distributing incoming traffic across multiple backend servers so that no single machine becomes the bottleneck.

Why load balancing exists — One server has a ceiling. Traffic doesn't care about your ceiling.

I learned this the hard way during my early days working on an e-commerce platform. We launched with a single production server. The first few weeks were smooth, but when a sale event started, traffic spiked. Within minutes, response times went from 200ms to 5 seconds. Users timed out. Orders failed. Revenue dropped. We scrambled to scale, but a single machine hitting CPU and memory limits doesn't magically get faster.

A single server has finite resources: CPU cores, RAM, network bandwidth, and disk I/O. Each request consumes some of these resources. Once you hit the ceiling, every new request either waits in a queue or gets rejected. This is where load balancing becomes essential.

By distributing traffic across multiple servers, you multiply your capacity. If one server handles 1,000 requests per second and you add three more identical servers behind a load balancer, you now handle roughly 4,000 requests per second. The load balancer sits in front of your backend servers and acts as a traffic director, deciding which server gets each incoming request.

The architecture looks like this: user sends a request to the load balancer (a single entry point), the load balancer examines the request and forwards it to one of the backend servers, the backend server processes it and sends the response back through the load balancer to the user. From the user's perspective, they're talking to a single server. Behind the scenes, their traffic is being distributed seamlessly.

Common algorithms — Choosing the right distribution strategy

Not all requests are created equal. Some are lightweight; others are compute-intensive. Not all servers are identical either. A load balancer needs a strategy to decide which server gets each request. Here are the most common approaches:

Round Robin — Requests cost about the same

Round robin cycles through your backend servers in order. Request 1 goes to Server A, request 2 to Server B, request 3 to Server C, request 4 back to Server A. It's simple and fair if every request takes roughly the same time to process.

Use round robin when your requests are relatively homogeneous. A real-time data dashboard where most API calls return lightweight JSON is a good fit. An image processing service where some requests process 1MB files and others process 100MB files? Not ideal.

Least Connections — Request costs vary

Instead of cycling through servers, the load balancer tracks how many active connections each backend currently has and routes the next request to whichever server has the fewest. This works well when requests have variable duration.

I used this approach on a ride-hailing platform where some user sessions were quick (book a ride) while others were long-lived (active driver tracking WebSocket connections). A driver's persistent WebSocket connection to Server A didn't prevent Server A from handling short HTTP requests, but it did count toward connection limits. Least connections ensured incoming requests found the server with breathing room.

IP Hash — Clients need the same backend

IP hash calculates a hash of the client's IP address to consistently route the same client to the same backend server. This is useful when you need session affinity: the user's session state lives on one server, and you want all their requests to hit that server so they don't lose context.

A drawback: if a backend server fails, all clients hashed to that server suddenly lose their session. Modern best practice is to store sessions in a distributed cache (Redis) instead, but IP hash still appears in legacy systems and certain compliance-heavy applications.

Weighted Round Robin — Servers differ in capacity

If your servers aren't identical—say, one server has 8 CPU cores while another has 4—you can assign weights. A server with weight 2 receives twice as many requests as a server with weight 1. This lets you squeeze more throughput from your better hardware without running identical instances.

I've used this in cost-conscious setups where we ran a mix of instance types on AWS: some large instances (weight 3) and some medium instances (weight 1). It balanced cost and performance.

Health checks — The balancer must know who is alive

A load balancer is worthless if it routes traffic to a server that's down. Imagine Server B crashes—it's no longer responding to requests, but the balancer keeps sending traffic to it anyway. Users hit timeouts, and your capacity just dropped by a third.

This is why health checks exist. The load balancer periodically pings each backend server with a simple request (usually a lightweight HTTP GET to a dedicated endpoint) and checks if it responds. If a server fails three health checks in a row, the load balancer marks it as unhealthy and stops routing traffic to it until it recovers.

I typically implement a health check endpoint that looks something like this:

app.get('/health', (req, res) => {
  const checks = {
    database: checkDatabaseConnection(),
    cache: checkRedisConnection(),
    diskSpace: checkDiskSpace(),
  };

  const allHealthy = Object.values(checks).every(c => c === true);
  
  if (allHealthy) {
    res.status(200).json({ status: 'healthy', checks });
  } else {
    res.status(503).json({ status: 'unhealthy', checks });
  }
});

The health check returns HTTP 200 if everything is fine, and 503 (Service Unavailable) if something is broken. The load balancer checks this endpoint every 10-30 seconds. When a server becomes unhealthy, traffic drains from it gracefully (no mid-request interruption) and the other servers absorb the load.

Health checks are not just about uptime; they're about meaningful availability. A server might be running but have no database connection. A health check catches this and prevents sending traffic to a server that will fail every request.

Layer 4 vs Layer 7 — TCP speed vs HTTP intelligence

Load balancers operate at different levels of the network stack, and this affects both performance and capability.

Layer 4 (Transport Layer)

L4 load balancers work at the TCP level. They make routing decisions based on IP protocol data (source IP, destination IP, port numbers) but don't look inside the HTTP request. An L4 balancer looks at the raw TCP connection and says, "New connection from 203.45.67.89 to port 443—send it to Server B," then pipes all bytes between client and server without inspection.

L4 is extremely fast because there's minimal processing. It's often implemented in hardware or kernel-level software. Use L4 when you need raw throughput and simplicity: non-HTTP protocols (WebSockets, gRPC, raw TCP), real-time systems with tight latency requirements, or high-frequency trading platforms.

Layer 7 (Application Layer)

L7 load balancers actually parse HTTP requests. They read the Host header, the URL path, the cookies, and the request body. This lets them make intelligent routing decisions:

• Route all requests to /api/admin to a smaller, specialized admin server
• Route requests with cookie user_tier=premium to high-performance servers
• Route requests based on hostname (host-based routing for multi-tenant systems)

L7 adds latency because the balancer must read and parse each request. But on modern systems, this overhead is negligible (microseconds), and the routing flexibility it provides is invaluable.

I used L7 routing on a SaaS platform where different customers' APIs lived on the same load balancer but separate backends. By reading the Host header, I routed customer-a.example.com to Backend A and customer-b.example.com to Backend B, maximizing cache efficiency and customer isolation.

Most cloud load balancers (AWS ALB, GCP Cloud Load Balancing, Azure Application Gateway) are L7-aware. Use L7 for HTTP/HTTPS services. Use L4 (Network Load Balancer) for extreme throughput or non-HTTP protocols.

Common mistakes — Pitfalls I've seen crash systems

• Ignoring session state. If you deploy with IP hash but don't replicate session data, a server failure orphans entire user cohorts. Lesson: store sessions externally in Redis or a database, not in application memory.
• Weak health checks. A health endpoint that only checks if the process is running isn't enough. Check dependencies: database, cache, external APIs. If a database connection is dead, the server should report unhealthy.
• No graceful shutdown. When you deploy a new version, servers need to finish in-flight requests before terminating. Without this, clients get connection resets. Use connection draining (AWS: Connection Draining; Nginx: graceful shutdown) to give requests time to complete.
• Mismatched algorithm choice. Using round robin on a service where requests vary wildly in cost means some servers are overloaded while others idle. Match the algorithm to your workload.
• Single point of failure. A single load balancer is a bottleneck. Use a highly available load balancer (cloud-managed services handle this) or run multiple balancers behind a DNS failover.

Key takeaways

• Load balancing distributes traffic across multiple backend servers, multiplying your capacity beyond what a single machine can handle.
• Choose your algorithm based on request characteristics: round robin for uniform requests, least connections for variable costs, IP hash for session affinity (though external session storage is better), weighted for heterogeneous servers.
• Health checks are critical; they let the balancer detect and isolate failing servers, preventing traffic from reaching broken backends.
• Layer 4 balancers are fast but dumb (IP and port only); Layer 7 balancers understand HTTP and enable intelligent routing based on paths, headers, and cookies.
• Real-world reliability comes from external session storage, robust health checks, graceful shutdown, and highly available load balancer infrastructure.