System Design Diary Page 3/30- Load Balancers
In our previous pages we have already explored the single server problem and the horizontal scaling architecture. We have seen how Single Point of Failure (SPOF) due to single server failing can cause disruption in the availability of our application.
To prevent SPOF, we often introduce multiple servers in our application right? But now the question arises, how do we orchestrate such that: the incoming requests from our clients knows which server instance to route to?
I mean we can obviously try using DNS routing, how? Let’s say we give our servers IP addresses. We then make sure that the DNS records point our given domain name to the IP addresses of our servers:
Sure this can work for sometime. Up till a point this is okay where we do not need to keep track of- the number of requests our servers are processing, the health checks for current state of our server (whether it has failed or still serving), whether a server instance is serving a user specifically (stateful sessions). If these do not matter to us in details, we can use DNS routing.
Let me clarify for concerns that may arise later: DNS routing can be customized or configured to have health checks (whether a server has failed or is actively running) but this may cause problems where the user let’s say already has the failed server IP address stored in their browser cache.
If the user goes to the cached IP address then they will be served from the currently inactive server. However if they start afresh the DNS resolver can resolve to a healthy server IP address. So keeping these in mind is necessary.
In summary, DNS-based routing can distribute incoming traffic across multiple server IP addresses using different strategies, such as round-robin or weighted routing which we will talk about later. However, DNS primarily performs the job of resolving domain names into IP addresses. It does not inherently provide the real-time workload awareness, request-level routing, or immediate failure handling that we might need when managing multiple application servers.
This brings us to a more sophisticated solution: Load Balancers.
What is a Load Balancer?
Before we know what specific problems a load balancer is meant to solve, let’s get to define what a load balancer can be:
- Software installed, running on a dedicated machine- e.g. we can install HAProxy or NGINX (we will talk about the differences between them in details in later pages, just know that both work as load balancer for web traffic). This software allows managing incoming request traffics to our servers.
- Cloud Service that runs on cloud infrastructures and manages our traffic e.g. AWS Elastic Load Balancing, Azure Load Balancer etc.
How does Load Balancer solve our problems?
In system design and architecture, note that every component is meant to solve one specific problem very well; cases where one of them fails, another one is created exactly to solve them! Just like how DNS routing was never meant to solve the traffic management system, we have our component exactly for solving this: Load Balancer!
Now, the DNS Resolver works along with the load balancer to retrieve the response from our server instance. Note, each component is designed for a specific purpose: DNS resolver for resolving our domain name to the load balancer’s IP Address while the load balancer directs our request traffic to one of our server instances.
What load balancer can do differently from DNS routing is: It can obtain information about its backend servers through mechanisms such as health checks, connection tracking, response-time measurements and, where configured, resource-utilization monitoring. These help answering the following questions-
- How many requests each server is currently dealing with
- How much time the servers recently took to give back response
- How much storage and RAM capacity each server has
- Are the servers even healthy?
This is specifically important because without knowing the current context, it will not be a smart choice to deploy a traffic managing system.
Let’s say our Server A is handling 5 requests right now and each request needs about let’s say 1.5 GB of RAM. Will it be a smart choice to occupy this server with a 6th request right now even though it seems to be the fastest responding server?
Now we may not know how much RAM our 6th request may occupy. However, through additional monitoring and resource-utilization metrics, our load-balancing system can identify that Server A is already under significant memory pressure and prefer a less occupied server instead.
Get Ishmaam’s stories in your inbox
Join Medium for free to get updates from this writer.
Without considering the available server capacity, a routing strategy that prioritizes response time alone could unintentionally direct more traffic towards an already overloaded server, in this case- Server A, potentially causing memory exhaustion and server failure.
Now, obviously, we could engineer DNS routing to perform some of these tasks as well. In fact, modern DNS providers already offer advanced traffic-management features such as health checks, weighted routing and geographical routing.
However, remember that each component serves a specific purpose. DNS primarily resolves domain names into IP addresses, whereas a load balancer is designed to distribute incoming network traffic across backend servers.
Even with advanced DNS configurations, we cannot achieve the same level of request-level traffic management because DNS decisions happen during domain-name resolution, not every time an application request arrives.
This is why a dedicated load-balancing component is a more suitable solution for managing traffic across multiple application servers.
What if now our own Load Balancer fails?
An interesting question that might hit your mind, “you talked about single server causing SPOF, what if our own load balancer is the one causing SPOF now?”
This is a very good engineering decision question! We are approaching in the right direction if architectural questions like this hits our mind. There are approaches we can take if this happens e.g:
- Cloud providers such as AWS and Azure offer managed load-balancing services designed to provide high availability.
For example, AWS Elastic Load Balancing can distribute traffic across multiple Availability Zones. The cloud provider manages the underlying load-balancing infrastructure, reducing the operational burden of maintaining redundant load balancers ourselves.
However, high availability still depends on the service configuration and the availability of the surrounding infrastructure. Using a managed load balancer does not automatically eliminate every possible single point of failure.
2. Use active-passive redundancy with a floating virtual IP.
In this approach, we have two load balancers: one active and one on standby.
Both are configured to use a shared floating virtual IP (VIP). Under normal conditions, the active load balancer handles incoming traffic, while the standby load balancer remains ready to take over.
If the active load balancer fails, a failover mechanism transfers ownership of the floating IP to the standby load balancer. Because the service-facing IP remains the same, clients generally do not need to resolve a new IP address through DNS.
However, failover may take some time, and this approach requires suitable network support and correct configuration.
3. Use active-active load balancers.
In this approach, multiple load balancers handle incoming traffic simultaneously instead of keeping one on standby.
Clients can reach these load balancers through DNS-based routing or another upstream routing mechanism that distributes traffic between them.
If one load balancer fails, the routing mechanism must detect the failure and direct traffic toward the remaining healthy load balancers. If DNS is used, cached DNS responses can still cause some clients to attempt connections to the failed load balancer.
This approach can increase traffic-handling capacity and avoid leaving a standby load balancer idle. However, it requires careful configuration to ensure that traffic is distributed correctly and that the remaining load balancers can handle traffic when one fails.
Conclusion
This article was just an overview of what a conventional load balancer is meant to do, we will delve into the load balancing algorithms like round robin, least response time, least connections, weighted versions of these algorithms in the next few pages. And while we are at it, we will also talk about AWS’s ELB, NGINX vs HAProxy and many other relevant load balancing topics into details.
