Wojdylak Michal·Just nowKV Cache: The Hidden Variable in LLM Inference BenchmarksWhen you load-test an inference engine, the number that decides the result is often not how fast its kernels are, but how much memory it…
Agent Native·2h agoContext Engineering Is Dead. Long Live Cache Engineering.What happens when we give the agent direct control over its active KV cache?A response icon1A response icon1
DhanushKumar·3h agoHow RadixAttention Reinvents LLM Caching and Memory ManagementRadixAttention is one of the important optimizations used in modern Large Language Model (LLM) inference systems. It was introduced by…
intelligentLearn·7h agoFrom symptoms to fixes: the trade-offs inside an LLM inference system — Part 1: Start with the…Where the real trade-offs live, worked through with numbers you can check.
System Design with Sage·1d agoHow LLM Batching Really Works Under Latency PressureI often see teams improve tokens per second, ship the change, and then get paged because first-token latency doubles under live traffic…
Siim Tolk·1d agoOne day, one GPU cluster each: NVIDIA’s inference-at-scale workshop at GTC BerlinGTC Berlin started for me a day before the keynote, with the full-day workshop Deploying and Optimizing AI Inference at Scale ( NVIDIA…
Siim Tolk·1d agoOne day, one GPU cluster each: NVIDIA’s inference-at-scale workshop at GTC BerlinGTC Berlin started for me a day before the keynote, with the full-day workshop Deploying and Optimizing AI Inference at Scale ( NVIDIA…
Agent Native·2d agoPure C inference engine to run MoE models (2.8T) on consumer hardware744-billion-parameter model on a laptop with 25 GB of RAM sounds absurd. The quantized checkpoint alone takes up hundreds of gigabytes.
intelligentLearn·2d agoWhat good inference looks like: promises and measurementsAn inference service is good when it keeps a stated promise to a stated kind of work at a known cost. Before anything is optimized: what is…
