We train AI Engineers. We deploy Enterprise AI. Join 100K+ AI practitioners at Towards AI Academy or reach out for help making your company AI native. towardsai.com
Your Second GPU Is Bought for the Cache, Not the Model
There are three reasons to add GPUs and they call for different answers. Then there is a split that has nothing to do with any of them: reading the prompt and writing the answer want different machines.
Everything so far has assumed one machine, and for most deployments that is the right assumption. This part is what happens when it stops being enough. One question comes first, because the answer decides what you buy. Are you adding GPUs because the model does not fit, because you need more throughput, or because one card cannot hold a long enough conversation? Three problems, three purchases. The figure below carries the…
We train AI Engineers. We deploy Enterprise AI. Join 100K+ AI practitioners at Towards AI Academy or reach out for help making your company AI native. towardsai.com
The cost accounting here is the most rigorous I've seen on this topic. Reading the baseline is most of the skill, and the DistServe/Mooncake/Dynamo comparison table alone is worth the read. One thing the piece prices in milliseconds and megabytes…
the cache-first framing is right, but it hides the tax: kv cache only helps when scheduling, prefix reuse, and gpu interconnect are designed together. otherwise the second gpu becomes an expensive memory shelf while pcie traffic and fragmented batches eat the latency gains everyone expected
Such a good idea to structure the runtime jargon into several smaller parts as you have. Lots of important and sometimes little-known terminology covered!
I wished though you had written those pieces more yourself or at least edited them for easier…
--
More from Satsawat Natakarnkitkul (Net) and Towards AI
