The Hidden Numbers That Decide How Many Users You Can Serve: Connection Pools, Worker Limits, and Load Balancers
The petrol station
A petrol station has four pumps. Cars arrive, fill up and leave, and life is good. Then one driver parks at a pump and walks off to buy snacks. The station now has three working pumps, and a queue is forming behind a car that is not even fuelling.
Your database is that station, and the pumps are connections. Each request needs one to talk to the database, and the number is limited. The driver who wanders off is a request that keeps its connection while it waits for something slow, such as a call to a payment partner.
The big idea: capacity hides in config files
Your app’s real capacity is rarely set by CPU. It is set by small numbers that nobody reviews:
- How many connections the database allows
- How big each app instance’s connection pool is
- How many workers or threads each instance runs
- How long the load balancer and your app keep idle connections open
If these numbers disagree with each other, the weakest one decides your ceiling.
Surprise 1: a bigger pool is often slower
The HikariCP documentation, the connection pool used by most Java applications, makes a striking point using a demonstration from Oracle’s Real-World Performance group. Reducing the pool from 2,048 connections to 96, with no other change, cut response times from about 100ms to about 2ms, more than a 50 times improvement. The wiki goes further and argues that even 96 is probably too many unless the database server has a lot of cores.
Why? A database can only do so much work at once. With far more active connections than CPU cores, the database wastes time switching between them. The best design is a small pool kept busy, with the rest of your requests waiting in line inside the application, where waiting is cheap.
The wiki’s starting formula, which it says has held up across benchmarks for years, is connections equals twice the database server’s core count plus the number of effective disk spindles. Two things to remember: it is a starting point to test, and it applies to the database, not to each app instance. Pool sizing is ultimately specific to your deployment.
You can also size from traffic using Little’s Law: connections in use equals queries per second times the seconds each query holds a connection. At 2,000 queries per second and 50 ms each, about 100 connections are busy. If the query slows to 500 ms, you suddenly need 1,000, and the pool becomes your bottleneck before the CPU does.
Surprise 2: autoscaling can take down your own database
Here is a failure that catches even experienced teams. Each instance of your app keeps a pool, so total connections equal the pool size times the number of instances.
- 4 instances with a pool of 50 each is 200 connections.
- A traffic spike triggers autoscaling to 40 instances, and you now ask for 2,000 connections.
- The database, already under stress, is hit by a wall of new connections.
PostgreSQL’s default max_connections is 100. SQL Server’s ADO.NET pool defaults to a maximum of 100 per connection string, and MongoDB drivers default to up to 100 open connections in the pool. So the multiplication happens quickly. Serverless functions make it worse, because each concurrent execution can open its own connection.
Fixes: put a connection pooler or proxy between the app and the database, keep per instance pools small, cap your maximum instance count according to your database limit, and give each service its own database role with a connection limit.
Get Rufus Kehinde’s stories in your inbox
Join Medium for free to get updates from this writer.
Your database, one by one
1. PostgreSQL: each client connection gets its own backend process, which is why connections are expensive and why the default limit is 100. PgBouncer is the most widely used pooler. It supports session pooling (the default), transaction pooling and statement pooling. Transaction mode gets the most out of few connections, but it breaks some session features, such as LISTEN/NOTIFY, and session level settings. Since PgBouncer 1.21, protocol level prepared statements are tracked per client, which fixed a long standing pain point, so check your version.
2. SQL Server: the ADO.NET pool keeps a separate pool for each distinct connection string, so small differences in a string can create extra pools. The default maximum is 100, and when it is reached, requests queue. If they wait too long you get the familiar error saying the timeout elapsed before a connection could be obtained from the pool. In Azure SQL Database, connection limits also depend on the service tier you chose.
3. MongoDB: drivers keep their own pool per client instance, with a default maximum of 100 connections, so many app instances multiply quickly. Set maxPoolSize deliberately, and watch the total across your fleet.
4. Oracle: sessions and processes are limited at the database, and Oracle offers connection pooling options such as Database Resident Connection Pooling and the Universal Connection Pool, so check what fits your architecture.
5. MySQL: the same idea applies, with a max_connections limit and proxies such as ProxySQL.
How pools go wrong
1. Leaks. Code that opens a connection and forgets to release it on an error path. Always release in a finally block or a using style construct.
2. Holding a connection while doing something slow. Never keep a connection or an open transaction while calling an external API. Get the data, release the connection, then call out.
3. Long transactions. In PostgreSQL, sessions stuck in an “idle in transaction” state hold connections and locks without doing anything.
4. Chatty code. The N+1 pattern from one of my previous posts here holds a connection for much longer than needed.
5. Long wait timeouts. If a request waits 30 seconds for a connection, everything behind it is stuck too. Make the connection acquisition timeout shorter than your request timeout, so you fail fast and visibly.
Watch these signals: connections in use, connections idle, and above all the number of requests waiting for a connection. In HikariCP with Micrometer, for example, a pending connections metric shows threads waiting.
Worker limits — three numbers that must agree. The traffic path has three limits in a row:
- Concurrent requests the load balancer will send to each instance
- Workers or threads in each instance
- Connections in the pool, which should be fewer than the workers
Typical examples: Node.js runs one event loop per process, so you run several processes. Python servers such as gunicorn use a configured number of worker processes. PHP-FPM has a maximum number of child processes. Java servers use thread pools. .NET uses its thread pool. Many workers share a small pool, and that is fine, as long as the waiting queue is bounded and has a timeout.
Load balancing in plain words
A load balancer is the traffic warden sending cars to pumps. Things worth knowing:
1. Layer 4 and layer 7. A layer 4 balancer forwards connections without reading them. A layer 7 balancer reads HTTP, so it can route by path or header and apply smarter rules.
2. Algorithms. Round robin is simple. Least connections adapts when some requests are slow. Weighted variants help when servers differ. Hash-based routing keeps a key on the same server. Some balancers also offer “power of two choices,” picking the better of two random servers, which works well at scale.
3. Health checks. A shallow check asks “is the process alive?” A deep check queries dependencies such as the database. Be careful with deep checks: if the database blips, every instance fails its check at the same time, the balancer removes them all, and a small problem becomes a total outage. Keep the load balancer’s check shallow, and use separate readiness and liveness endpoints in orchestration systems.
4. Graceful deploys. When an instance is removed, it should stop accepting new requests, finish current ones, then exit. Load balancers have a draining or deregistration delay for this, and orchestrators have a termination grace period, which in Kubernetes defaults to 30 seconds. Make sure the app handles the shutdown signal.
5. Stateless apps. Avoid sticky sessions where you can, and keep sessions in a shared store, so any server can serve any user.
The timeout trap behind random 502 errors
This one confuses even experienced engineers. An AWS Application Load Balancer keeps idle connections to your servers open for 60 seconds by default. Node.js closes idle connections after 5 seconds by default. In the unlucky moment when a new request arrives just as the server closes the connection, the load balancer sends the request into a dead connection and reports a 502 Bad Gateway, even though your app is healthy.
The rule: the backend’s keep-alive timeout must be longer than the load balancer’s idle timeout. For a 60-second balancer, a common setting is 65 seconds. In Node.js, also keep headersTimeout higher than keepAliveTimeout. The same logic applies in other stacks, such as the — keep-alive option in gunicorn. A good general ordering is that the client’s idle timeout is shortest, then the load balancer, then the backend.
The scale lens
- The connection arithmetic. 50 per instance times 40 instances is 2,000 connections against a default limit of 100. At the same time, 2,000 queries per second at 50 ms is only about 100 busy connections. The danger is not usually the work; it is the number of idle and wasted connections.
- Connection storms. After a deploy, a database failover, or an outage, every instance reconnects at once. Use backoff with jitter on reconnects, stagger rolling deployments, and avoid opening a full pool at startup.
- Pools hide slow queries until they do not. A query that gets ten times slower needs ten times the connections, so a slow query can look like a connection problem.
Retries multiply. Load balancer retries, client retries, and application retries stack up. Keep one layer in charge of retries.
- Capacity plan the whole chain, from load balancer to workers to pool to database, with load tests, and know which limit breaks first.
A security note
- Least privilege per service. Give each service its own database credentials with only the permissions it needs, and cap connections per role where the database supports it.
- Encrypt in transit. Use TLS between the load balancer, the app, and the database, and do not expose the database or the pooler to the public internet.
- Be careful with forwarded headers. Many apps read the client’s IP from headers set by proxies. If your app trusts them from anywhere, attackers can fake them and bypass IP-based limits; see more about that here. Only trust them from your known proxies.
- Keep health endpoints minimal. Do not reveal versions, dependency names or internal details to the public.
- Slow connection attacks. Short timeouts for headers and bodies protect your workers from clients that hold connections open on purpose.
- Rotate credentials safely. Set a maximum connection lifetime in your pool, so rotated credentials take effect without a restart.
The tradeoffs
Poolers and proxies add a hop, a component to run, and sometimes restrictions, such as transaction mode limits.
Small pools make requests wait, which is intentional, but you must set sensible timeouts and monitor the queue.
Aggressive health checks improve failover speed but risk false alarms and flapping.
Long keep-alive timeouts use more resources on the server.
A limit that is too tight wastes hardware, and one that is too loose lets a spike take everything down.
Tools to start with
- Pool metrics from your library, such as HikariCP with Micrometer, ADO.NET performance counters, and driver metrics
- pg_stat_activity in PostgreSQL and the equivalent session views in SQL Server, Oracle, and MongoDB, to see who holds connections
- Poolers and proxies: PgBouncer, ProxySQL, and the managed database proxies your cloud provider offers
- Load balancer metrics: target response time, 5xx counts, connection counts
- Load tests that scale instance counts to catch the connection multiplication before production does
In your stack. Check each one’s documentation for your version:
- Java: HikariCP
- .NET: ADO.NET pooling and EF Core’s DbContext pooling
- Node.js: pg Pool, Prisma’s connection limit setting, and the pool options in TypeORM and Sequelize
- Python and Django: persistent connections with CONN_MAX_AGE, and native pooling in recent Django versions
- PHP and Laravel: PHP-FPM keeps a connection per process, so use a pooler such as PgBouncer or ProxySQL for large fleets
- Go: SetMaxOpenConns and its relatives in database/sql
- MongoDB drivers: maxPoolSize
Sometimes the biggest limits are not in your code. They are in the numbers nobody reviews.
