Member-only story
Your KV Cache Is Bigger Than Your Model
Four multiplications tell you what one conversation costs in GPU memory: 4.5 GiB at 128K on gpt-oss-120b, 40 GiB on the dense shape most people still picture.
13 min readAug 16, 2026
This is Part two of seven. The layer where capacity planning actually happens.
- Part 1: Start Here: The Words Everyone Uses About LLM Inference
- Part 3: What a Kernel Is, and Why Everyone Is Writing New Ones
- Part 4: Quantization Is Four Decisions, Not One
- Part 5: Tuning vLLM: What Every Setting Does to the Arithmetic
- Part 6: Your Second GPU Is Bought for the Cache, Not the Model
- Part 7: Your Context Length Decides What a Kernel Is Worth
Part one ended on a problem it could not solve. Batching lets many users share one read of the model’s weights, which is what makes serving affordable, but the part of the model that remembers your conversation is yours alone, and no amount of batching spreads that cost.
This part is about how big that private memory actually gets. Capacity plans come apart here more than anywhere else in the stack, and they fail the same way every time: the model fits, the demo is fast, and then real…
