8 ms in the Notebook, 4 Seconds in Production: Serving a Model as an API (AI Engineering in 30 Days, Day 4)
Why a model that predicts fast can serve slowly, how dynamic batching fixes it, and how to budget every step against a p99 target.
A team ships a recommendation model. In the notebook, model.predict(x) returns in 8 milliseconds. They wrap it in a quick Flask app, put it behind a URL, and it works for the first ten users. Then real traffic arrives. At a couple of hundred requests a second, answers take 4 seconds and keep climbing, and the GPU shows 12% utilization. Nothing is broken. The model is fine. The serving is not.
That gap between a model that predicts and a model that serves is Day 4 of the series: turning a model file into an API.
Answering Day 3’s practice question
Yesterday ended with exactly that story: 8 ms in a notebook, 4 seconds behind Flask at about 200 requests a second, and the GPU only 12% busy. What is wrong?
- The app handles one request at a time. New requests wait in a queue, and when they arrive faster than they leave, the queue grows every second. That waiting is where the 4 seconds come from.
- Each GPU pass carries a single input. Between passes, the GPU waits while the worker does the rest of that one request.
- A GPU is built for batches. The lesson’s key fact: a GPU running a batch of 32 inputs takes almost the same wall-clock time as running one, because it does the math in parallel. Feed it one at a time and most of what you pay for sits idle.
The fix: dynamic batching
Requests come from different users at different moments. You cannot ask them to arrive in neat groups of 32. Dynamic batching makes the groups on the server: it holds each request for a short, bounded window, collects whatever else arrives, and runs them all as one batch.
The window has two limits racing each other: a maximum batch size (32 in the lesson’s example) and a maximum wait (5 ms). Whichever comes first closes the window and fires the pass. You spend a few milliseconds of latency to buy many answers per pass. Tune it wrong either way and you lose: too short and the batches stay tiny, too long and the wait eats your latency budget.
Get Roni Das’s stories in your inbox
Join Medium for free to get updates from this writer.
You rarely write this yourself. Model servers such as BentoML, Triton and TorchServe do it. In BentoML it is a few arguments on the API method: batchable=True, max_batch_size=32, and a maximum latency. The lesson also loads the model once at startup, not on every request, which is another thing quick Flask apps often get wrong.
Run each replica near the knee
Batching sets your position on a curve. As you push more concurrent requests through one replica, throughput climbs until the GPU saturates, then flattens while latency shoots up. In the lesson’s figure, throughput levels off near 2,500 requests a second while p99 latency goes from about 15 ms to 300 ms. Run each replica near the knee of that curve, and add replicas for more capacity, rather than pushing one replica past it.
Budget every step against the p99 target
The lesson’s example gives the whole path a 120 ms p99 target and assigns every step a number:
- network in: 8 ms
- TLS, auth, rate limit: 4 ms
- cache lookup: 1 ms
- batch wait: 5 ms
- preprocessing: 6 ms
- GPU forward pass: 28 ms
- postprocessing: 4 ms
- serialize and network out: 12 ms
- total: 68 ms
That leaves 52 ms of headroom. The GPU pass is the biggest single line, at 41% of the time used, so that is where to cut first: a smaller or quantized model, or better batch fill. Tuning the network while a 28 ms pass dominates is polishing the wrong step.
Two common mistakes
- Watching the average. Quality is the tail: p50, p95, p99. A page that makes 5 model calls has about a 1 minus 0.99⁵, or roughly 5%, chance that at least one lands on the p99 slow path. Heavy users hit it many times a session.
- Scaling to zero for a model people wait on. A fresh replica has to pull a multi-gigabyte image, start the runtime and load the weights onto the GPU. For a large model that is 30 to 90 seconds in which it serves nothing. Keep a warm floor, and scale on latency headroom before p99 breaks, not after.
Practice for Day 5
A fraud model worked well at launch. Three months later it misses more fraud, and nobody changed the code. What changed, and what would you automate so the team finds out before customers do? We work through it at the start of Day 5, which is about training pipelines that retrain themselves.
Where this comes from
Every number in this post is from the free model serving lesson linked as the source. AI Engineering in 30 Days runs next to System Design Interview in 30 Days, which today covers sharding a chat app’s messages. Both courses, AI Engineering and System Design, are on systemdesign.academy. You can read the model serving lesson itself here.
