Load Balancers: How One Website Can Handle Thousands of Requests at Once…
1. The problem it solves
Picture this scene… “it’s results day at your college, and the official website is about to publish examination marks. Thousands of students are hammering the refresh button on the same page at the same time. Every one of them expects to see their result right away.”
When a website relies on just one server, that single machine must process every incoming request. But any server has a ceiling — limited CPU capacity, finite memory, restricted network throughput, and only so many connections it can juggle at once. As the volume of traffic climbs, replies slow down. Push it far enough and requests start timing out, or the site goes dark entirely.
Throwing money at a bigger server can ease the strain, but it doesn’t cure everything. A machine can still die from a failed component, a software crash, or a network fault. And if that lone server goes down, the whole application disappears with it.
So the real challenge isn’t simply “how do we make one server faster?” It’s “how do we spread incoming requests across several servers while keeping the application responsive and available?”
This is precisely the job a load balancer does. Think of it as a traffic controller sitting between users and a pool of application servers. Rather than each visitor connecting to a single machine, every request flows through the load balancer, which picks a suitable server to handle each request or connection.
2. What existed before
Older Solution I: A single server
The most basic setup runs the entire application on one machine. Users connect to it directly, and it copes with everything — web pages, business logic, and sometimes even database work.
This design is simple to build and easy to maintain. For a small college project or a website with a handful of visitors, it may be perfectly adequate.
But its weaknesses surface the moment traffic grows. The server has a fixed pool of resources. When requests arrive faster than it can serve them, they queue up, and response times balloon.
It also introduces a single point of failure. In other words, one breakdown can take the entire service offline. There is no second application server standing by to take over.
Older solution II: Vertical scaling
Vertical scaling means beefing up the existing server rather than adding more machines. A developer might, for instance, swap a box with 4 CPU cores and 8 GB of RAM for one with 16 cores and 64 GB.
This can boost performance, since the application gets more computing muscle. It’s also fairly straightforward — the application may not need significant architectural changes.
Yet vertical scaling hits hard limits. You can’t upgrade a machine indefinitely, and top-tier hardware gets expensive fast. Upgrades may force downtime. Most critically, the system still hinges on a single server, so the risk of hardware failure never goes away.
Alternative: DNS-based distribution
DNS — the Domain Name System — translates a domain name into an IP address. A DNS service can hand out different server addresses for the same domain name, thereby spreading users across several servers.
This can help scatter traffic, but DNS is not the same thing as a request-by-request load balancer. DNS answers may be cached by clients and resolvers. The application doesn’t necessarily have immediate control over where each request lands, and DNS on its own doesn’t check whether a server is currently healthy.
DNS-based distribution can complement load balancing — especially for spreading traffic across locations — but it cannot replace all the functions a load balancer provides.
3. Why the new solution is preferred
A load balancer flips the approach from “make one server do everything” to “run multiple servers and divide the work.”
The user connects to a single shared entry point. Behind it sit several application servers. The load balancer picks a server according to a routing policy and can steer new traffic away from unhealthy machines.
This improves scalability, because extra servers can be added as traffic grows. It also improves availability, because the failure of one server need not bring down the whole application.
There is, however, a trade-off. The load balancer itself must be reliable and capable of handling the traffic. If only one load balancer is deployed and it fails, it becomes yet another single point of failure. Production systems therefore typically rely on redundant or managed load-balancing infrastructure.
The key idea is that a load balancer doesn’t magically make every server faster. It helps the application put several servers to efficient use. Algorithms and data structures determine how the traffic controller decides where each request should go.
4. How it works
At a high level, a user sends a request to the load balancer. The balancer inspects its available servers, chooses one according to a policy, and forwards the request. The selected server does the work and sends a response back. Health checks and server-state tracking help the balancer avoid sending traffic to machines that are out of service.
Step 1: The request reaches the load balancer
Suppose a user opens college.example.com. The browser connects to the application's public entry point — the load balancer.
The load balancer receives the request or connection and identifies the relevant service. Depending on its type, it may use information such as IP addresses, ports, HTTP paths, or headers.
Layer 4 load balancing works mainly with transport information such as TCP or UDP. Layer 7 load balancing understands application-level information such as HTTP requests and can route based on paths or headers. Google Cloud documents both Layer 4 and Layer 7 load balancing. (See Reference 1.)
Step 2: The balancer selects a server
The balancer needs a policy for choosing a backend server. A backend is a server that performs the actual application work.
Common policies include:
- Round Robin: send requests to servers in a repeating order.
- Weighted Round Robin: send more requests to servers assigned higher weights.
- Least Connections: prefer the server with the fewest active connections.
- IP Hash: use a hash of the client’s IP address to select a server.
NGINX documents Round Robin, Least Connections, and IP Hash as supported load-balancing methods. (Reference 2.)
Round Robin is simple, but it assumes servers and requests are reasonably similar. Least Connections can be more useful when some requests take longer than others.
Step 3: The selected server processes the request
The load balancer forwards the request to the chosen backend. The application server processes it — perhaps querying a database or generating a page.
The load balancer generally does not perform the application’s main work. Its job is to decide where the work should happen and to manage the traffic path.
Step 4: The response returns to the user
The backend produces a response. Depending on the architecture, the response may travel back through the load balancer, which returns it to the client.
From the user’s point of view, the website still looks like one application, even though several servers may be working behind the scenes.
Step 5: Health checks monitor servers
A health check is a periodic test used to determine whether a server is ready to receive traffic. It might check whether a port is reachable or whether an application endpoint responds correctly.
If a backend becomes unhealthy, the load balancer can stop directing new requests to it. AWS Elastic Load Balancing documents health-based routing to registered targets. (Reference 3.)
This does not guarantee that every request succeeds. A failure can occur mid-request, and health checks may take time to detect a problem. But they reduce the chance of repeatedly sending new traffic to a server that is already unavailable.
Data structures behind server selection
A load balancer must store server information and make selection decisions quickly. The exact internal implementation depends on the product and configuration. The structures below explain common ways to implement the algorithms; they should not be taken as a claim that every commercial load balancer uses these exact structures.
1. Array or list: Round Robin
A list can store the available servers:
[Server A, Server B, Server C]
The balancer also keeps an index pointing to the next server. After selecting one, it advances the index. When it reaches the end, it wraps around to the beginning.
Get Wvinisha’s stories in your inbox
Join Medium for free to get updates from this writer.
With three servers, requests may be assigned as follows:
- Request 1 → Server A
- Request 2 → Server B
- Request 3 → Server C
- Request 4 → Server A
The server list occupies O(n) space for n servers. Selecting the next server can take O(1) time when the index is maintained.
A plain list works well because Round Robin doesn’t need to search for the best server. It only needs to remember the next position.
2. Min-heap: Least Connections
A heap is a tree-based structure that keeps the smallest or largest value at its root. A min-heap keeps the smallest value at the top.
For Least Connections, each server can be represented by its current number of active connections:
- Server A: 12 connections
- Server B: 5 connections
- Server C: 9 connections
A min-heap could place Server B at the root because it has the smallest connection count.
In a standard binary min-heap:
- Finding the minimum is O(1).
- Inserting a server or changing its priority usually takes O(log n).
- Removing the minimum takes O(log n).
This makes a heap useful when the system frequently needs to select the least-loaded server.
Important implementation detail: connection counts change whenever requests begin or finish. A heap must be updated correctly after these changes. Some implementations instead scan server counters or use specialized structures. A heap is a conceptual approach, not a claim about the private implementation of AWS or Google.
3. Hashing and consistent hashing
A hash function converts an input, such as a user ID, into a number. Consistent hashing is a technique that places both servers and keys on a circular range of hash values, often called a ring.
A key is assigned to the next server encountered when moving clockwise around the ring. When a server is added or removed, only a portion of keys typically need to be reassigned, rather than redistributing every key.
This is valuable in distributed systems where server membership changes frequently. The original consistent-hashing research was designed to reduce the amount of data that must move when nodes change. (Reference 4.)
Consistent hashing is not necessarily the default policy for every load balancer. It is particularly useful when maintaining affinity between a key and a backend matters, such as cache placement.
Worked example: 12 requests and 3 servers
Suppose a college portal has three servers, each initially healthy:
- Server A
- Server B
- Server C
Assume Round Robin is configured and every request is similar in processing cost…
The balancer starts with Server A:
Each server receives four requests. That’s a fair distribution if the servers have equal capacity and requests require similar work.
Now imagine Server B becomes unhealthy. Once the health check detects the problem, the balancer removes B from the eligible server list. Subsequent requests are shared between A and C.
If the system kept using the old list without health checks, it could keep sending traffic to B, producing failed requests. This example shows why server selection and health monitoring must work hand in hand.
What if we used the wrong data structure?
If the balancer used a basic list and searched every server to find the least-connected one, selecting a server could take O(n) time. That may be acceptable for a small number of servers, but it becomes less efficient as the backend pool grows.
A heap can reduce the cost of selecting the smallest value, but it adds complexity because connection counts change. A heap that isn’t updated correctly may select a server that is no longer the least loaded.
Similarly, using ordinary hashing without consistent hashing may cause many keys to be reassigned when servers are added or removed. Consistent hashing reduces this movement, although it adds ring management and may need virtual nodes to improve distribution.
The right structure depends on the goal: simple fairness, current load, or keeping users and data associated with particular servers.
5. Real world use cases
1. Amazon Web Services: Elastic Load Balancing
AWS Elastic Load Balancing distributes incoming traffic among registered targets such as EC2 instances, containers, or IP addresses. Its documentation explains how Application Load Balancers select targets and how health checks support traffic routing. (Reference 3.)
This is useful for web applications that need to handle changing traffic while avoiding unhealthy backend targets.
2. NGINX
NGINX can act as an HTTP load balancer in front of multiple application servers. Its documented methods include Round Robin, Least Connections, and IP Hash. (Reference 2.)
For example, a Node.js application could run several instances behind NGINX rather than exposing only one process directly to users.
3. Google Cloud Load Balancing
Google Cloud provides Application Load Balancers for HTTP(S) traffic and Network Load Balancers for transport-level traffic. Its documentation describes global and regional configurations, backend selection, and integration with different compute services. (Reference 1.)
This lets applications distribute requests across backends and, depending on the configuration, across regions.
4. A student’s everyday example: College portal
A college portal may be used by students to check attendance, notices, fees, and examination results. During result announcements, many students may open the same page at once.
A load-balanced design could distribute those requests among multiple application servers. The actual architecture of a particular college website should not be assumed unless it is publicly documented; this is an illustrative example.
When load balancing may not be a good fit…
A small website with very little traffic may not need several servers or a separate load-balancing layer. Adding one too early can increase cost and operational complexity.
Load balancing also cannot fix every bottleneck. If all application servers depend on one overloaded database, distributing requests among more servers may simply increase pressure on that database.
6. Conclusion
A load balancer helps applications cope with growing traffic by distributing requests across multiple servers. Compared with relying on one powerful machine, this approach can improve scalability and availability. Algorithms such as Round Robin and Least Connections, supported by suitable data structures, help decide where requests should go. The best design depends on workload, server health, cost, and the application’s requirements.
References :
- Google Cloud. “Cloud Load Balancing Overview.” Primary source — official documentation.
https://docs.cloud.google.com/load-balancing/docs/load-balancing-overview - NGINX. “Using nginx as HTTP Load Balancer.” Primary source — official documentation.
https://nginx.org/en/docs/http/load_balancing.html - Amazon Web Services. “How Elastic Load Balancing Works.” Primary source — official documentation.
https://docs.aws.amazon.com/elasticloadbalancing/latest/userguide/how-elastic-load-balancing-works.html - David Karger et al. “Consistent Hashing and Random Trees: Distributed Caching Protocols for Relieving Hot Spots on the World Wide Web.” Research paper — primary source.
https://people.csail.mit.edu/karger/Papers/web.pdf - NGINX Documentation. “HTTP Load Balancing.” Primary source — official documentation.
https://docs.nginx.com/nginx/admin-guide/load-balancer/http-load-balancer/
