1 Million Users ≠ 1 Million Requests Per Second: Understanding the Numbers Behind System Design
Why understanding workload, capacity, and bottlenecks matters more than simply adding more servers.
When I started studying system design, one of the things I found difficult wasn’t understanding the technologies themselves.
I knew what load balancers did. I understood why applications use Redis, database connection pools, background workers, and multiple backend instances.
But there was something I couldn’t quite connect.
How do engineers actually know when they need these things?
For example, imagine you’re in a system design interview and the interviewer asks:
“Design a banking application that supports one million users.”
You could immediately start drawing an architecture involving a load balancer, several backend services, Redis, PostgreSQL, and a message queue.
But how do you know whether you need two backend instances or ten?
How many database connections should your application maintain?
At what point does your database become a bottleneck?
And how do you know whether your existing infrastructure can handle the expected traffic?
These questions led me to an important part of system design: capacity planning and back-of-the-envelope estimation.
The more I studied it, the more I realized something.
Scaling isn’t simply about adding infrastructure. It’s about understanding the workload, measuring resource consumption, and identifying where a system reaches its limits.
Let’s break that down.
1. One Million Users Doesn’t Mean One Million Requests Per Second
This was one of the first distinctions I had to understand.
When someone says an application has one million users, that doesn’t tell us how much traffic the application is receiving.
Not every registered user is active.
And not every active user is making requests continuously.
To estimate the actual workload, we need to establish a few assumptions.
Let’s imagine we’re building a banking application with:
- 1,000,000 registered users
- 200,000 daily active users (DAU)
- 30 API requests per active user per day
These are hypothetical values, not measured production statistics.
Calculating daily requests
If 200,000 users each generate 30 API requests daily:
200,000 × 30 = 6,000,000 requests/day
Our application handles approximately six million requests every day.
But servers process requests continuously, so we need a more useful measurement.
Requests per second (RPS).
There are 86,400 seconds in a day.
Therefore:
Average RPS = 6,000,000 ÷ 86,400 ≈ 69.4 RPS
That’s approximately 69 requests per second.
Think about that.
An application with one million registered users could average only 69 backend requests per second under these assumptions.
Of course, user activity isn’t distributed evenly throughout the day.
People might use the application more frequently in the morning, during lunch, or after work.
This is why we also calculate peak traffic.
Estimating peak requests per second
Suppose we assume peak traffic reaches five times the daily average.
Peak RPS = 69.4 × 5 ≈ 347 RPS
Now we have a more useful initial estimate.
Instead of designing infrastructure based on one million registered users, we’re designing around an assumed peak of approximately 347 requests per second.
The 5× multiplier is only a starting assumption. In a production environment, we’d examine actual traffic patterns, including short-lived spikes, rather than treating it as a universal rule.
The takeaway: user count doesn’t determine infrastructure requirements. User behavior does.
2. Requests Per Second and Concurrent Requests Are Different
Here’s where things get more interesting.
Suppose our backend receives 500 requests per second.
Does that mean the server is processing 500 requests simultaneously?
Not necessarily.
We need to consider how long each request remains in the system.
This introduces a concept called Little’s Law.
For a stable system:
Average concurrent requests = Throughput × Average response time
Imagine:
- Throughput: 500 requests/second
- Average response time: 200 milliseconds (0.2 seconds)
Our average number of requests in flight is:
500 × 0.2 = 100 concurrent requests
Now suppose the database becomes slower and average response time rises to two seconds.
Assuming the system still completes 500 requests per second:
500 × 2 = 1,000 concurrent requests
Notice something interesting.
The throughput hasn’t changed.
But the average number of requests in flight has increased from 100 to 1,000.
This is important because requests remaining in the system longer can increase pressure on memory, connection pools, and worker concurrency limits.
If the application cannot sustain that level of concurrency, requests may queue, time out, or fail.
There is another distinction worth mentioning.
Concurrent requests don’t necessarily mean concurrent CPU execution.
An asynchronous Python application can have hundreds of requests waiting for database or network responses while only a small amount of CPU is actively being used.
Conversely, a CPU-intensive request might occupy significant processing capacity throughout its execution.
So understanding concurrency requires understanding what each request actually does.
3. How Many Requests Can One Backend Server Handle?
This is another question I initially struggled with.
You might hear statements like:
“A server can handle 1,000 requests per second.”
But that number means very little without context.
Consider these endpoints:
GET /health
Returns a small JSON response.
GET /transactions
Reads transaction history from PostgreSQL.
POST /transfer
Validates a transfer, updates database records, and potentially communicates with other services.
These endpoints have different resource requirements.
The first might require very little processing.
The second depends heavily on database query performance.
The third might involve multiple queries, transactions, locks, and external dependencies.
Therefore, their maximum throughput could differ significantly even when running on the same hardware.
A simplified CPU capacity calculation
Imagine we have a backend instance with four vCPUs.
For estimation purposes, assume it provides four CPU-seconds of execution capacity every second.
Suppose each request consumes an average of 10 milliseconds of CPU time.
That’s 0.01 CPU-seconds per request.
The theoretical CPU-limited throughput becomes:
4 ÷ 0.01 = 400 RPS
At this rate, the application would theoretically consume all available CPU capacity.
But operating at 100% utilization leaves very little room for unexpected spikes.
Suppose we target 70% CPU utilization.
400 × 0.7 = 280 RPS
That gives us an estimated planning capacity of 280 requests per second per instance.
For our banking application with an estimated peak of 347 RPS:
347 ÷ 280 ≈ 1.24
Rounding up, we’d need at least two instances under these simplified CPU assumptions.
However, two instances wouldn’t necessarily provide the redundancy or additional capacity required for an actual production system.
And here’s the important part.
This calculation only considers CPU.
It doesn’t tell us whether our database, RAM, network, or other dependencies can support the same throughput.
That’s why theoretical calculations are only the starting point. Load testing is how we validate them.
4. Adding Backend Instances Can Create Database Problems
Let’s say our banking application initially runs on two backend instances.
Each instance maintains a PostgreSQL connection pool with a maximum of 20 connections.
Our application could potentially open:
2 × 20 = 40 database connections
Now imagine traffic increases and we scale from two backend instances to ten.
If each maintains the same pool configuration:
10 × 20 = 200 database connections
We’ve increased the potential number of database connections fivefold.
But did we increase the database’s capacity?
No.
We’re still connecting to the same PostgreSQL instance.
In fact, too many concurrent database connections can increase resource consumption and contention.
Get Aghahowa Jeffrey’s stories in your inbox
Join Medium for free to get updates from this writer.
This helped me understand why horizontal scaling isn’t always straightforward.
Scaling your application servers can simply move the bottleneck to another component.
Estimating database connection requirements
Suppose our application receives 347 RPS.
Each request performs an average of two SQL operations.
That gives us:
347 × 2 = 694 database operations per second
Now assume each operation holds a database connection for 20 milliseconds on average.
Using Little’s Law:
694 × 0.02 = 13.88
That’s approximately 14 simultaneously occupied database connections on average.
But what happens if query or connection-hold time increases to 200 milliseconds?
694 × 0.2 = 138.8
Now approximately 139 connections would be occupied on average if the system sustained the same operation rate.
Suddenly, a connection pool configured for much less concurrency could become exhausted.
Requests begin waiting for available database connections, and latency increases.
This is why monitoring database query duration, transaction time, pool wait time, and connection utilization is so important.
A tool like PgBouncer can help manage database connections, but connection pooling doesn’t magically increase the database’s processing capacity.
Sometimes the real solution is optimizing slow queries, adding appropriate indexes, reducing unnecessary operations, or upgrading database resources.
5. Why CPU Utilization Alone Doesn’t Tell the Whole Story
Let’s imagine our application starts experiencing slow response times.
Our first instinct might be to check CPU utilization.
But suppose CPU usage is only 35%.
Does that mean the backend has plenty of capacity?
Not necessarily.
The application could be waiting on:
- Slow database queries
- An exhausted connection pool
- External API responses
- Disk I/O
- Network connections
- Locks or resource contention
In other words, a system can be overloaded even when CPU utilization is relatively low.
This is why performance monitoring needs to go beyond a single metric.
For backend services, useful measurements include throughput, CPU usage, memory consumption, error rate, and p95 or p99 response latency.
For PostgreSQL, we might also monitor query duration, active connections, lock waits, IOPS, and transaction throughput.
For background workers, we’d examine processing rates, queue depth, and the age of pending jobs.
Different metrics help us identify different bottlenecks.
And until we know which resource is limiting performance, adding infrastructure is largely guesswork.
6. Background Workers Need Capacity Planning Too
Another component that becomes important as applications grow is the background worker.
Imagine users make transfers, and our system sends notifications afterward.
Rather than keeping the transfer API waiting for notification delivery, we might publish notification jobs to a queue.
Workers consume those jobs asynchronously.
But how many workers do we need?
Let’s assume:
- 120 notification jobs arrive per second
- Each job takes 500 milliseconds on average
- Each worker process can handle 10 concurrent jobs
First, calculate the average processing slots required:
120 × 0.5 = 60 concurrent jobs
If each worker supports 10 concurrent jobs:
60 ÷ 10 = 6 workers
Six workers represent the theoretical minimum under those assumptions.
But operating exactly at maximum capacity leaves no room for bursts or processing variations.
If we target 70% concurrency-slot utilization:
60 ÷ (10 × 0.7) ≈ 8.57
Rounding up gives us nine worker processes.
Again, these numbers depend on the type of work.
If jobs are CPU-intensive, configuring more concurrent jobs per process may not improve throughput.
If they’re primarily waiting for external network responses, asynchronous concurrency can be very effective.
The objective is to understand how much work arrives, how much processing capacity exists, and whether workers can keep up.
Otherwise, the queue will continue growing.
7. How Do You Know When Your System Actually Needs Scaling?
At this point, the question changes.
Instead of asking:
“How many servers should I add?”
We should be asking:
“Which component is approaching its sustainable capacity, and what’s causing it?”
Imagine we’ve load-tested a system and obtained these hypothetical results:
ComponentSustainable capacityCurrent demandLoad balancer5,000 RPS347 RPSBackend fleet560 RPS347 RPSDatabase reads800 ops/sec500 ops/secDatabase writes150 ops/sec100 ops/secBackground workers200 jobs/sec120 jobs/sec
At first glance, the system has spare capacity.
But suppose traffic increases by 50%, with the workload mix remaining constant.
Database write demand becomes:
100 × 1.5 = 150 writes/second
That reaches the database’s assumed sustainable write capacity.
Meanwhile, the load balancer is still operating far below its own limit.
If database write operations begin slowing down, scaling the load balancer won’t help.
Adding more backend instances may not help either.
The appropriate response depends on the actual bottleneck.
We might investigate database locks, expensive transactions, write amplification, disk performance, or opportunities to optimize the data model.
That is what capacity planning helps us understand.
The system’s overall performance is constrained by its bottlenecks, not by how many technologies are included in its architecture.
8. The Numbers Are Estimates, Not Absolute Truths
One thing I think is important to emphasize is that these calculations don’t give us perfect answers.
They give us a starting point.
In an interview, estimates help us reason about architectural choices and explain our assumptions.
In production, we need measurements.
For example, we can use load-testing tools such as Locust or k6 to simulate increasing traffic.
We can then observe what happens at 50, 100, 200, 500, or 1,000 requests per second.
Does CPU usage rise proportionally?
At what point does p95 latency increase sharply?
Do database connections become exhausted?
Does throughput stop increasing even as we send more requests?
Does the error rate begin to rise?
The answers tell us more about the system’s actual capacity than any arbitrary server-sizing rule.
We also need headroom for traffic bursts and failures.
If a system requires three instances to serve peak traffic, we may need additional capacity to tolerate an instance going offline.
And if the application takes time to launch new instances, autoscaling cannot necessarily protect us from sudden spikes.
All of these considerations contribute to responsible capacity planning.
9. The Mental Model I’m Taking Away
The biggest shift in how I now approach system design is that I try to reason from the workload backward.
Instead of immediately choosing technologies, I start by asking:
1. How many users are active?
Not just the total number of registered accounts.
2. What are those users doing?
How many requests, transactions, database queries, or background jobs does each action generate?
3. What is the expected peak workload?
What happens during busy periods or sudden spikes?
4. How expensive is each operation?
How much CPU time, memory, network bandwidth, and database capacity does it consume?
5. What can each component sustainably handle?
Not just theoretically, but within acceptable latency and error limits.
6. Where is the bottleneck?
Which resource becomes saturated first, and what happens to the rest of the system when it does?
7. What is the most appropriate response?
Do we optimize, cache, queue, vertically scale, horizontally scale, or redesign part of the system?
These questions provide a much stronger foundation for architectural decisions.
Final Thoughts
Studying system design has been teaching me that understanding distributed systems isn’t just about knowing when to use Redis, Kafka, Kubernetes, PostgreSQL, or a load balancer.
It’s about understanding the behavior of the application we’re building.
Every request has a resource cost.
Every component has a limit.
And changing one part of a system can affect the performance of several others.
Adding more backend instances might improve application throughput, but it could also increase pressure on the database.
Caching might reduce database reads, but it introduces consistency and invalidation challenges.
Adding background workers might clear a growing queue, but only if downstream services can handle the additional processing rate.
These are the kinds of trade-offs that make system design interesting.
And while I’m still learning, one principle has become much clearer to me:
Before choosing an architecture, understand the workload. Before scaling a component, identify the bottleneck.
Because building a scalable system isn’t about adding more servers.
It’s about understanding where the system reaches its limits, why those limits exist, and how to address them intelligently.
What metrics do you typically examine first when investigating a backend performance bottleneck? I’d love to learn how other engineers approach this in production.
