Load Balancing and Stateless Service Design — A Practical Guide

Load balancing gets talked about as if it were the hard part of scaling a service horizontally. It isn't — distributing requests across servers is a solved problem with well-understood algorithms. The actual hard part, the one that determines whether adding more servers helps at all, is making sure your service doesn't depend on state that only exists on one specific server.

Why Statelessness Comes Before Load Balancing

Imagine a server that stores a user's shopping cart in memory after login.

Add a second server behind a load balancer, and this breaks immediately: a user's cart-add request might land on server A, and their next request — checking out — might land on server B, which has never seen their cart. No load balancing algorithm fixes this; the problem is architectural, not routing. The fix is moving cart state out of process memory and into a shared store every instance can read.

Now any server instance can serve any request for any user, because the state that matters lives outside the process. This is the actual definition of a stateless service — not "has no state," but "holds no state a later request depends on that isn't externally accessible."

Load Balancing Algorithm 1: Round-Robin

Round-robin distributes requests to backend servers in a fixed rotating order.

It's simple and works well when requests take roughly the same amount of time to process. It breaks down when request costs vary widely — a server that happens to get a run of expensive requests keeps receiving new ones at the same rate as an idle one, since round-robin has no concept of current load.

Load Balancing Algorithm 2: Least Connections

Least-connections routes each new request to whichever backend currently has the fewest active connections.

This adapts to uneven request costs far better than round-robin, since a server bogged down with slow requests naturally receives fewer new ones until it catches up. It's the better default for most real APIs, where request duration varies (a search query versus a health check, for example).

Load Balancing Algorithm 3: Consistent Hashing

Consistent hashing routes requests based on a hash of some request property (often a user or session ID), so the same key consistently lands on the same backend — useful for cache locality without full statefulness.

The distinction from sticky sessions matters: consistent hashing is a routing optimization (so a user's requests tend to hit a server that already has their data warm in a local cache), not a correctness requirement. If that server goes down, the request routes elsewhere and still works correctly — it's just a cache miss, not a broken session. This is the difference between hashing as an optimization and stickiness as a crutch for state that should have been externalized.

Why Sticky Sessions Are a Trap

Sticky sessions configure the load balancer to always route a given user to the same backend server, typically because that server holds the user's session in local memory.

This looks like it solves the statelessness problem, but it only hides it. Three real consequences follow: if that specific server crashes, every user stuck to it loses their session entirely. Load distribution becomes uneven over time, since some servers accumulate more "sticky" long-lived users than others, regardless of current load. And scaling down — removing a server during a deploy or traffic dip — forcibly evicts every session pinned to it. Fixing the actual problem (externalize the session to Redis or a database) removes the need for stickiness entirely, and is almost always less work long-term than operating around it.

Health Checks: Making the Load Balancer Aware of Failure

None of the above matters if the load balancer keeps sending traffic to a server that's crashed or degraded.

The load balancer polls this endpoint on an interval and stops routing new requests to any instance that fails enough consecutive checks, resuming once it recovers. A meaningful health check verifies the dependencies that actually matter to serving requests (database connectivity, for instance) — a health check that only confirms the process is running will report "healthy" even when the server can't actually do its job.

Key Takeaways

Load balancing algorithms are the easy part — round-robin for uniform request costs, least-connections for variable ones, and consistent hashing when you want cache locality without correctness depending on it. The precondition that makes any of this work is a genuinely stateless service, where session and application state live in a shared store rather than server memory; sticky sessions are a sign that precondition hasn't been met, not a valid substitute for it. And a load balancer without real health checks will happily keep sending traffic to a dead server, so health checks that verify actual dependencies (not just process liveness) are not optional in production.