Quantization Is Four Decisions, Not One

FP8, INT4 and NVFP4 are only one of them. The accuracy penalties people quote are often not in the source at all.

Satsawat Natakarnkitkul (Net)

11 min read

·

Aug 20, 2026

--

1

Listen

Share

Press enter or click to view image in full size

Part four of seven. Same layer as the KV cache, for the same reason: both are about bytes.

• Part 1: Start Here: The Words Everyone Uses About LLM Inference
• Part 2: Your KV Cache Is Bigger Than Your Model
• Part 3: What a Kernel Is, and Why Everyone Is Writing New Ones
• Part 5: Tuning vLLM: What Every Setting Does to the Arithmetic
• Part 6: Your Second GPU Is Bought for the Cache, Not the Model
• Part 7: Your Context Length Decides What a Kernel Is Worth

Parts one and two both ended in the same place: you are limited by how many bytes you move, and the cache is eating your capacity. Quantization is the obvious response. Store the numbers in fewer bits, move fewer bytes, fit more conversations. It does work.

Press enter or click to view image in full size

It is also barely specified, and two people in the same conversation often mean different things by it. “Just quantize it”…