Why Every Good System Learns to Say “Full”: A Practical Guide to Rate Limiting, Backpressure and Load Shedding
The conductor
A bus conductor in a busy city has one job that matters more than any other: knowing when to say “full.” A bus that takes every passenger who shows up does not carry more people. It breaks its axle, and nobody gets to work.
A conductor who says “full” is not being unkind. He is the reason the bus keeps moving.
Your app has the same problem. When more traffic arrives than the system can handle, you have two choices. You can accept everything and fail slowly for everyone, or you can say no quickly to some requests and keep serving the rest.
These terms are often mixed up, so here they are in plain words
1. Rate limiting: a rule about how many requests one client may make in a period of time, for example 10 per second.
2. Quotas: a larger allowance over a longer period, for example 100,000 requests per month on a pricing plan.
3. Load shedding: when the whole system is under pressure, deliberately dropping low priority work so critical work survives.
4. Backpressure: when one part is full, it signals the parts feeding it to slow down, instead of letting work pile up without limit.
Did you know that most time, your worst client is usually you?
When people think about limits, they imagine attackers. In practice, the biggest sources of overload are friendlier and closer to home:
- Your own mobile app retrying in a loop after an error
- A partner’s script with a bug
- Your own service hammering another one after an outage
- A well meaning customer running a huge export at peak time
And there is a second surprise: where a request comes from is a weak identity. On mobile networks, many real customers can share one IP address, so an IP based limit can block innocent people, while IPs are also easy to fake. OWASP’s guidance for API protection recommends configuring limits on other keys, such as tokens or fingerprints, where possible. Limit by who the caller is, such as the API key, account or user token, and use IP only as an extra signal.
The algorithms, in plain words
- Fixed window. Count requests in each time slice, such as 100 per minute, and reset at the boundary. It is simple, but it allows a burst at the edge: 100 requests at the last second of one minute and 100 in the first second of the next, so 200 in two seconds.
- Sliding window. Spreads the counting across the boundary and smooths that burst, at the cost of more bookkeeping.
- Token bucket. Each client has a bucket that refills at a steady rate, say 10 tokens per second, up to a maximum of 20. Each request spends a token. A client who has been quiet can burst 20 requests, then settles at 10 per second on average. Stripe documents that it uses a token bucket for its request rate limiter.
- Leaky bucket. Smooths traffic into a steady outflow, useful when a downstream system needs an even pace.
- Concurrency limiter. Instead of counting requests over time, it caps how many are in flight at once. This suits expensive endpoints, because the cost is how long each request holds a worker.
Stripe’s four layers
Stripe published how it protects its API in production, and the design is a good template:
1. A request rate limiter (token bucket), which Stripe calls by far the most important and recommends starting with.
2. A concurrent requests limiter, to control CPU intensive endpoints.
3. A fleet usage load shedder, which reserves a share of capacity for critical requests, so non-critical ones are rejected when the fleet is busy.
4. A worker utilization load shedder, the last line of defense, which starts dropping lower priority traffic when a machine is under heavy pressure.
What to return when you say no
- HTTP 429 Too Many Requests when a client exceeded its limit, or 503 Service Unavailable when the server itself is overloaded.
- A Retry-After header, which tells the client how long to wait. HTTP defines it for 429 and 503 responses.
- Optional limit headers. An IETF draft proposes the RateLimit-Limit, RateLimit-Remaining and RateLimit-Reset fields. It was still a draft in the versions I found, and many APIs use their own names, so document whichever you choose.
- A clear message. Say which limit was hit, so the developer on the other side can fix it.
Get Rufus Kehinde’s stories in your inbox
Join Medium for free to get updates from this writer.
Where to enforce limits
Use several layers, from the outside in:
- The edge: a CDN or WAF handles huge volumes and abusive patterns, including volumetric attacks, before they reach you.
- The gateway or reverse proxy: NGINX, Envoy, Kong, and the cloud API gateways, for per-key limits.
- The application: business aware limits, such as per tenant, per plan and per endpoint.
- The workers: concurrency caps on queue consumers, so they cannot overload the database.
- The database itself: the last wall.
Protect the database directly
A limit at the front does not help if one expensive query can still flatten your database. Every serious database offers ways to cap the damage:
- PostgreSQL: a connection pooler to cap connections, and statement_timeout to stop runaway queries.
- SQL Server: Resource Governor, to limit how much CPU and memory a workload can take.
- Oracle: Database Resource Manager, for the same purpose.
- MongoDB: maxTimeMS to cap query time, and connection pool limits in the drivers.
- All of them: set sensible pool sizes.
Backpressure in practice
- Bound your queues. An unbounded queue does not remove the overload, it only postpones the crash and hides the problem. When the queue is full, reject new work visibly.
- Cap prefetch and concurrency. Consumers should take only as much work as they can handle, as with RabbitMQ’s prefetch limit. Pull based systems, such as Kafka consumers or SQS pollers, are naturally paced by the consumer.
- Use streams that respect pressure. Node.js streams, for example, signal when a buffer is full so the producer pauses. Reactive Streams libraries on the JVM and .NET include backpressure for the same reason.
- Set timeouts at every hop, and pass deadlines along, so a request nobody is waiting for any more does not keep consuming resources.
Good manners for clients
The best limit is one that clients respect:
- Obey Retry-After.
- Use exponential backoff with jitter, so clients do not return together.
- Keep a retry budget. If every failed request is retried three times, your load can multiply by four at exactly the moment you are weakest.
- Use circuit breakers to stop calling something that is clearly down.
Rolling out limits safely
Stripe’s own lessons are a good checklist:
- Hook limiters into the middleware stack safely, so a bug in the limiter cannot break the API.
- Show clear errors to users.
- Build a switch to turn limiters off, for emergencies.
- Dark launch each limiter, which means running it in log-only mode first to see what it would have blocked before it blocks anything.
- Design the limiter so its own failure does not take the API down. Decide in advance what happens if the counter store, such as Redis, is unreachable. Usually, fail open for ordinary traffic, but think carefully for login and payments.
Counting across many servers
With many servers, the count must be shared. A common approach is a Redis based token bucket, where an atomic script updates the bucket in one step. Stripe’s published example does exactly this. Tradeoffs: a shared store adds a network call and a dependency, so some teams use approximate local limits that sync periodically. Use server time for refills, not client clocks, and remember that counters cost memory: 10 million active keys at roughly 100 bytes each is about 1 GB. (Illustrative numbers.)
In your stack
Check each one’s documentation for your version:
- ASP.NET Core: built in rate limiting middleware since .NET 7, with fixed window, sliding window, token bucket and concurrency limiters
- NestJS and Express: @nestjs/throttler and express-rate-limit
- Laravel: RateLimiter and the throttle middleware
- Django: the throttling classes in Django REST framework
- Spring and Java: Bucket4j and Resilience4j
- Go: the x/time/rate package
- Gateways and edge: NGINX limit_req, Envoy, Kong, AWS API Gateway and WAF rate based rules, Azure API Management policies, Cloudflare rate limiting
The scale lens
- Per user limits do not protect the total. 10 requests per second per user sounds safe, but 1 million users each using it is 10 million per second in theory. You need a global ceiling too.
- Set the ceiling from tested capacity. If 20 servers each handle 500 requests per second, your capacity is 10,000. A common starting point is to leave headroom, for example a ceiling around 70 to 80% of what a load test proves. Treat it as a starting point, not a law.
- Tiers and fairness. Free, standard and enterprise plans get different limits, and one noisy tenant should not starve others.
- Retry amplification. Three retries per failure can multiply load fourfold during an incident.
- Measure. Track how many requests get limited, who gets limited, and which limits are near their ceiling. A rising 429 rate is a signal, not just a statistic.
- Cost. Limits also cap cloud bills. OWASP’s own example describes a monthly bill jumping from about US$13 to US$8,000 because no limits or cost alerts existed.
A security note
Rate limiting is one of the most important security controls an API has. OWASP lists it as API4:2023, Unrestricted Resource Consumption.
- Login and password reset: limit by account and by IP, add progressive delays, and combine with multi factor authentication to resist brute force and credential stuffing.
- SMS and OTP endpoints: without limits, attackers can trigger huge numbers of messages, which costs real money.
- Enumeration and scraping: limit endpoints that reveal whether an account or record exists, and cap pagination sizes.
- GraphQL: attackers can batch many operations into one request to slip under limiters that count HTTP requests. Limit query depth and complexity, not only request counts.
- Payload and upload sizes: set maximums.
- Volumetric DDoS: application limits cannot stop floods that fill your network. Use your cloud provider’s or CDN’s DDoS protection.
The tradeoffs
- Limits can block legitimate users, especially on shared networks or during genuine spikes.
- Tight limits frustrate good customers, and loose limits protect nothing, so tune them from real traffic.
- Distributed counting adds a dependency and a failure mode.
- Load shedding means deciding which requests matter more, which is a business decision, not only a technical one.
- Backpressure pushes the problem to the user, so design clear messages and graceful behavior.
Tools to start with
- Your gateway’s or framework’s built in rate limiter, from the list above
- A load testing tool such as k6, to find your real capacity
- Log-only or dark launch mode for new limits
- Dashboards for 429 rates, limit utilization, queue depth and saturation
- Alerts for cost and abnormal usage
Saying no early is a feature, not a failure.
