System Design Interview: How Do You Keep One Slow Dependency From Taking Down Your API?
A practical way to budget time, bound retries, and keep optional work from blocking the core request.
Your API can be healthy on its dashboard while the page in front of a customer is still spinning. One dependency has started taking two seconds to answer, and every request that waits for it is holding a connection, a worker, or a place in a queue.
That is the little design problem I want to practise today. Not how to add another box to the architecture diagram, but how to stop one slow box from borrowing resources from everything around it.
Imagine a product page that needs the product record and also asks a recommendation service for optional suggestions. Product details are the promise the page has to keep. Recommendations are useful, but the page can still work without them. If both calls share one unbounded wait, a slowdown in recommendations can make the whole page feel broken.
This is a hypothetical example, not a report of a real outage. The useful part is the way we reason through it, one request at a time.
First, trace one request instead of drawing a bigger system
Start at the user action and follow the work until the response comes back. For this page, the path might be browser → API → product database, with a second call from the API to recommendations.
Now ask which parts are required for a useful response. The product name, price, and availability may be essential. Recommendations may not be. That distinction changes the design more than choosing between a queue and a cache too early.
Next, follow the waiting. If the API accepts 120 requests per second and each request can sit for two seconds, Little’s Law gives us about 240 requests in flight during steady state, before counting retries or other overhead. That number is only an illustrative estimate, but it makes the risk visible: slow downstream work can occupy upstream capacity even when the API’s own code is doing very little.
When the dependency gets slower, more requests remain in flight. If the server runs out of workers or connections, otherwise healthy requests start waiting too. Google’s SRE guidance describes this kind of positive feedback as a common shape of cascading failure: local overload pushes more load onto the remaining healthy capacity.
The first useful interview question is therefore not which component we should add. It is where requests wait, what each wait consumes, and which work can be skipped without breaking the user’s main task.
Give the whole request a deadline
Timeouts work best when they fit inside an end-to-end budget. Setting every downstream call to wait for five seconds does not make a five-second user experience acceptable; it just hides the time being spent at each hop.
Suppose the product page has an illustrative total budget of 900 milliseconds. We might reserve time for the API, the essential product lookup, optional recommendations, and the response back to the browser. The exact numbers depend on the product and its latency goal, so they should come from an explicit budget rather than habit.
Each child call should receive only the time that remains in the parent deadline. If the product lookup already used most of the budget, the API should not start an optional recommendation call that cannot finish in time. Likewise, it should stop work when the caller has disconnected, where the stack supports cancellation.
Timeouts are not simply a way to make errors happen faster. They bound how long scarce resources remain tied up. Set them too generously and a growing queue can consume the service; set them too aggressively and normal network variation may turn healthy requests into failures. The right value is something to measure and revisit, not copy from a blog post.
A retry is extra traffic, so make it earn its place
Retries can help when a failure is brief, but they can also send more work to a service that is already struggling. Before retrying, ask whether the operation is safe to repeat, whether the error is likely to be temporary, and whether enough of the request deadline remains to make another attempt useful.
For a read-only lookup, one carefully bounded retry may be reasonable. For a payment or other state-changing operation, the caller needs an idempotency strategy before retrying can be considered safe. A timeout does not prove the original operation never happened.
Get Emily’s stories in your inbox
Join Medium for free to get updates from this writer.
If a retry is justified, use backoff with jitter and a maximum attempt count or elapsed-time limit. Avoid retrying at every layer, because three layers each making three attempts can multiply the work reaching the dependency. AWS’s guidance on timeouts, retries, and backoff with jitter explains why retries need limits and why synchronized clients can create a fresh traffic spike.
Put a fence around the waiting room
Timeouts limit how long one request waits. They do not automatically cap how many requests wait at once. A system can still accumulate a large queue before its timers fire, so I would also ask how many concurrent calls and queued requests each dependency is allowed to consume.
That is where bounded queues, concurrency limits, and bulkheads help. A bulkhead keeps one kind of work from using every worker or connection needed by other work. A circuit breaker can temporarily stop calls after repeated failures, then allow a small number of probes to check whether the dependency has recovered.
None of those controls repairs the dependency. Their job is to keep the failure contained while the system chooses a sensible response. For example, a product page may return product details without recommendations. A checkout may need to fail clearly if a required inventory check cannot complete, rather than quietly pretending stock is available.
That distinction is important because graceful degradation should preserve a truthful result. Optional content can disappear for a while; a critical decision should not be guessed just to keep the page green.
Make the fallback part of the product decision
Engineers sometimes treat the fallback as a technical detail that can be picked after the architecture is finished. In practice, it changes what the user sees, and that choice belongs alongside the product requirement.
For each dependency, write down what the customer should receive when it is slow, unavailable, or returning questionable data. Maybe the API serves a cached recommendation set with a clear freshness limit. Maybe it omits recommendations entirely. Maybe a critical endpoint returns a retryable error and explains what the user can do next.
Then check the trade-off. A stale value can be better than no value for one product and dangerous for another. A fallback should have an owner, a freshness rule, and a way to tell that it is being used. Otherwise, a temporary path can silently become the normal one.
Test the slow case, not only the red case
The easiest failure to imagine is a dependency that returns an error. Slow responses are often more revealing because they keep connections and workers occupied while the dependency technically remains reachable.
I’d test a few concrete situations in a staging environment or a controlled fault-injection test. First, delay the dependency just beyond its normal latency and see whether the caller respects its deadline. Next, return transient errors and confirm the retry budget stays bounded. Then stop the dependency and check whether unrelated endpoints still have capacity. Finally, restore it and look for a retry burst or a queue that takes too long to drain.
Useful signals include dependency latency percentiles, timeout rate, retry rate, in-flight requests, queue depth, circuit-breaker state, and the success rate of the customer-facing operation. A dependency’s health check alone will not tell you whether users can complete the task that depends on it.
Google’s chapter on addressing cascading failures is a useful deeper read because it connects overload, resource exhaustion, and failure spreading across components.
A compact way to explain the design
When this comes up in a system-design conversation, I’d keep the explanation anchored to one request. Clarify the user-visible promise, trace each call, separate required work from optional work, and set a deadline the entire path can respect. Then explain which requests may retry, how waiting capacity is bounded, what the user sees when a dependency stays slow, and which metrics would reveal the failure before it spreads.
That sequence gives the interviewer something concrete to challenge. They can make the dependency slower, ask what happens after a retry, or change the endpoint from optional content to a critical write. Each follow-up tests whether the design still matches the product promise.
If you want another practice prompt with the same kind of failure reasoning, try designing device monitoring over an unreliable public network. It asks you to reason about late and repeated messages, unreachable devices, and what operators should see when the network goes quiet.
The next time a diagram looks perfectly healthy, follow one request through its waits. That is often where the real design starts.
~ Emily
