Serving Kimi-K3: a 2.8-trillion-parameter open agent model with 8xB300 GPUs
Kimi K3 on 8× NVIDIA B300 with SGLang and FlashInfer, for under $200: the memory math, how the model is split across GPUs, what the interconnect does, and what one node can actually serve.
Geodesic engineering · follow-up to Serving a capable open-source agent model on GPUs you can actually rent (Qwen3.5–27B)
In our last post, we served Qwen3.5–27B, a capable open agent model, on one A100 and one H200. This time we went to the other end of the open-model spectrum: Kimi K3, Moonshot AI’s 2.8-trillion-parameter model, which at its release was the strongest open-weights model on public leaderboards and competitive with the best closed models on agentic work.
The question we wanted to answer is practical. Can a small team stand up the state-of-the-art open model on hardware it can rent by the hour, measure it properly, and do it all for under $200? The answer is yes, on a single 8-GPU node, with some care about memory and about how the model is split across the GPUs. Everything below comes from one profiling run; every number is either measured on that run or computed from published specifications, and the computed ones are marked as estimates.
The short version. One 8× B300 node holds all 1.56 TB of Kimi K3’s weights and serves it at 104 tokens/s to one user (8.7 ms per token, 239 ms to the first token) and about 2,000 tokens/s across 64 concurrent agent sessions. Concurrency is capped at 53 running requests, set not by the KV cache (room for 4.8 million tokens) but by the fixed recurrent state of Kimi’s linear-attention layers. The whole project, including two failed provider attempts, cost about $49.
Why Kimi K3
Kimi K3 is a 2.8T-parameter mixture-of-experts model with about 104B parameters active per token, released by Moonshot AI in July 2026 with open weights under the Kimi K3 License. On the Artificial Analysis Intelligence Index it scored 57, which made it the leading open-weights model when its weights came out, ahead of GLM-5.2 (51) and DeepSeek V4 Pro (44) and comparable to Claude Opus 4.8 and GPT-5.5, behind only Claude Fable 5 and GPT-5.6 Sol. Nathan Lambert called it “clearly the strongest open model ever released” and “a watershed moment because frontier open-weight models are now real.”
For us the interesting part is the agentic results. Kimi K3 is built to work in loops of reasoning and tool calls, and that is where it competes with the closed frontier:
- Terminal work. 88.3 on Terminal-Bench 2.1, within a point of GPT-5.6 Sol (88.8) and Claude Fable 5 (88.0). This is the benchmark our team uses for agent training, so it matters to us directly.
- Tool use and the web. 91.2 on BrowseComp, 94.5 on MCPMark-Verified, 84.2 on MCP-Atlas, 76.5 on Toolathlon-Verified: Kimi K3 is at or near the top of every tool-calling benchmark in the table.
- Long-horizon software engineering. 42.0 on SWE-Marathon, the highest in the comparison, and 67.5 on DeepSWE.
- Computer use. 84.8 on OSWorld-Verified.
Thinking is always on in the API, with a reasoning_effort of low, high or max; the scores above are at max. In our own serving tests we turned thinking off and checked tool-call formatting on 37 prompts at temperature 0: Kimi K3 picked the right tool with the right arguments on 36 of 37.
The model, as the serving system sees it
Three things about the architecture decide how it has to be served:
- Almost all of the weight is in experts. 92 MoE layers × 896 experts × (3 × 3584 × 3072) = 2.72T parameters, exactly what the checkpoint’s metadata reports. Each token visits 16 experts per layer, so it reads only 1.8% of them.
- The experts are 4-bit by design. Moonshot trained them with quantization-aware training in MXFP4 (4-bit E2M1 values, two per byte, one shared 8-bit power-of-two scale per 32 values: 4.25 bits per weight) with MXFP8 activations. Serving in 4-bit is the original model, not a lossy conversion. The 2.72T expert parameters take 1.45 TB; the 57.2B BF16 parameters take 114 GB; the checkpoint is 1.56 TB.
- Two kinds of attention. 24 layers use Gated MLA (multi-head latent attention), which caches one compressed 576-value latent per token per layer. The other 69 layers use KDA (Kimi Delta Attention), a linear attention that keeps a fixed-size recurrent state per request instead of a growing cache. That state turns out to be the most important number in this whole post.
A $200 budget and the hardware decision
The weights alone are 1.56 TB. An 8× H200 node has 8 × 141 GB = 1.13 TB, so on Hopper Kimi K3 needs two nodes (16 GPUs), with the model split 16 ways across an InfiniBand link and the 4-bit experts unpacked in software, because Hopper has no FP4 hardware. A public report of that setup measured 17 tokens/s for one user. An 8× B300 node has 8 × 288 GB = 2.3 TB: the whole model fits on one machine, every GPU talks to every other over NVLink, and Blackwell multiplies FP4 natively.
Getting the node was the hard part. On-demand 8× B300 was sold out at our usual provider in all three regions where we had quota, and so was on-demand 8× H200. Modal had B300s immediately, at $7.10 per GPU-hour ($56.80/hour for the node), but its GPU containers are preemptible. We took that trade and engineered around it, which we describe at the end. Our budget was $200; with a planned run of about an hour of node time, that left room for one or two failures. We needed it.
Precision: what is stored in which format
DataFormatWhyRouted expert weights (2.72T)MXFP4 (4.25 bits)trained this way; Blackwell FP4 tensor coresExpert activationsset by the FP4 MoE kerneltrained with MXFP8; SGLang’s default kernel settingAttention, shared experts, dense layer, lm_head (57B)BF16left unquantized in the checkpointMLA KV cacheBF16 (default)FP8 E4M3 is an option; see the kernel resultsKDA recurrent stateFP32 (default)--mamba-ssm-dtype bfloat16 would halve it
We left the cache formats at SGLang’s defaults for this run so that the results describe the recipe SGLang verified for B300. As we will see, the state format is the most promising thing to change next.
GPU memory, to the gigabyte
SGLang reserves memory once at startup: weights first, then a KV pool and a state pool from what is left under --mem-fraction-static. Here is the arithmetic, from the log of GPU 0:
- Weights: 1.56 TB ÷ 8 ≈ 195 GB per GPU (measured 195.32 GB). Every GPU holds 1/8 of every matrix.
- MLA KV cache: one token costs 24 layers × 576 values × 2 bytes = 27.6 KB across all MLA layers. With decode context parallelism (below) each GPU stores 1/8 of every request’s tokens. SGLang allocated 601,664 tokens per GPU (15.5 GB), so the node holds 4.81 million tokens.
- KDA state: 69 layers × 96 heads × 128 × 128 values × 4 bytes ≈ 434 MB per request for the whole model. Each GPU keeps 1/8 of the heads, about 52 MB per slot, and SGLang allocated 265 slots (13.9 GB).
Now the catch. With the prefix cache on, SGLang keeps 5 state slots per running request: the live state plus snapshots at cached prefix boundaries, so that a later request sharing the prefix can resume from it. 265 ÷ 5 = 53 requests, and the server’s log says it plainly: “max_running_requests is capped to 53 by the mamba state cache.”
This is the same lesson we learned with Qwen3.5–27B on a 40 GB A100, where Gated DeltaNet state capped us at 9 requests. At 2.8T parameters and 2.3 TB of GPU memory, the hybrid-attention design moves the bottleneck from the KV cache to the recurrent state, and the default memory split leaves the KV pool mostly empty. Three settings would raise the cap, none of which we had budget to measure: a higher --mem-fraction-static (35 GB per GPU sat idle), --mamba-radix-cache-strategy extra_buffer_lazy (4 slots per request instead of 5), and --mamba-ssm-dtype bfloat16 (half the bytes per slot).
How the model is split across eight GPUs
There are four standard ways to split a model for inference, and SGLang’s verified B300 recipe uses two of them:
- Tensor parallelism (TP8) splits every weight matrix eight ways. Attention’s 96 heads become 12 per GPU; each of the 896 experts has its 3072-wide hidden layer split into 384 columns per GPU. Every GPU works on every token, and after each attention block and each MoE block the eight partial results are summed with an all-reduce.
- Decode context parallelism (DCP8) applies only to the MLA cache during decoding. Instead of every GPU keeping a full copy of every request’s latent cache (what plain TP8 does, since MLA has one shared latent per token), each GPU keeps 1/8 of each request’s tokens and computes all 96 heads over its slice. The partial attention outputs are merged with a log-sum-exp reduction. This is what multiplies the KV pool by eight.
- Expert parallelism (whole experts per GPU, with an all-to-all exchange of tokens) was off:
ep_size=1. On one NVLink node, splitting every expert eight ways keeps the load perfectly balanced and avoids all-to-all traffic. - Pipeline parallelism (layers 1–46 on some GPUs, 47–93 on others) was off too. It is the tool for multi-node setups, where it replaces bandwidth-hungry TP traffic over slower links.
What about FSDP? Fully sharded data parallelism is a training technique: it shards weights between data-parallel replicas and gathers them layer by layer for each step. An inference server keeps the weights resident and split by TP or EP instead, because re-gathering 1.56 TB of weights per forward pass would be far slower than serving it.
The communication, in numbers
With TP8 there are about two all-reduces per layer, so about 186 per forward pass. Their size depends on how many tokens are in the batch:
PhaseTokensOne all-reduce (tokens × 7168 × 2 B)What limits itDecode, 53 requests53760 KBlatency (a few µs each)Prefill chunk16,384235 MBbandwidth (~0.46 ms each at 900 GB/s, ring estimate)
For decoding, the messages are small and the cost is the number of round trips: ~186 × ~5 µs ≈ 0.9 ms per token, an estimate, against a measured 8.7 ms per token. For prefill, a 16K-token chunk moves roughly 186 × 0.46 ms ≈ 85 ms of all-reduce traffic per GPU in the ring estimate, a real share of the ~650 ms we measured per 16K tokens of prompt.
Get Will Youzhi Zou’s stories in your inbox
Join Medium for free to get updates from this writer.
The server log shows which collectives were used. SGLang’s own custom all-reduce ran with symmetric memory (a 48 MB buffer every GPU can address) and NVLink multicast, so small all-reduces become a single write into the switch fabric that is reduced and broadcast in the switch. SGLang’s Kimi K3 path also fuses the all-reduce with the following residual add and RMSNorm, saving a round trip to memory per layer, and NCCL 2.30.7 handled the remaining collectives. FlashInfer provides the same kind of fusion as a library call (flashinfer.comm.allreduce_fusion: all-reduce, residual, RMSNorm and optional FP8/FP4 quantization in one kernel); in this run SGLang's own fused path was active and the FlashInfer fusion flag was off.
Where the time goes in one token
A surprise in this budget: although 98% of the parameters are 4-bit experts, at batch 1 most of the bytes come from the 2% in BF16, because every token reads all of them (14.0 GB per GPU) but only 16 of 896 experts (3.2 GB). As the batch grows, more distinct experts are touched each step, and by a few dozen requests nearly all of them are, which is where FP4 earns its keep.
The serving setup
The server is SGLang 0.5.23 (the lmsysorg/sglang:v0.5.23-cu130 image: torch 2.14.1, CUDA 13, FlashInfer 0.7.2) with SGLang's verified B300 "Balanced" recipe plus our metrics and limits:
sglang serve --model-path /models/Kimi-K3 --trust-remote-code \
--tp-size 8 --dcp-size 8 --mem-fraction-static 0.85 \
--reasoning-parser kimi_k3 --tool-call-parser kimi_k3 \
--context-length 65536 --max-running-requests 128 \
--weight-loader-prefetch-checkpoints --weight-loader-prefetch-num-threads 1 \
--enable-metrics --enable-cache-reportMost of the important choices are ones SGLang made for us, and its log records them:
- MoE:
flashinfer_mxfp4, FlashInfer's TensorRT-LLM-generated FP4 block-scale MoE kernel. At startup it autotunes 7 kernel profiles for this model's shapes, which took 3 minutes 47 seconds. - Attention: MLA prefill with FlashInfer’s
trtllm_mlakernels, MLA decode withcutedsl_mla(a CuTe DSL kernel that supports DCP). Both need 64-token (or 32-token) pages, so the paged KV cache uses 64-token pages. - Prefix cache: on. 94.9% of agent prompt tokens were served from it.
- Chunked prefill: 16,384 tokens per chunk, so a long prompt is processed in slices between decode steps.
- CUDA graphs: captured for decode batches 1, 2, 4, 8, 12, … up to 53 (74 seconds, 0.5 GB). Prefill CUDA graphs are disabled for KDA models.
- Weight loading: 1.56 TB from a network volume. SGLang’s default multi-threaded loader reads every shard on every one of the 8 ranks, 12.5 TB in total, and our first attempt was on track for an hour. With
--weight-loader-prefetch-checkpointseach rank reads 1/8 of the shards into the shared page cache: 6.8 minutes. With autotuning and graph capture, the server was ready 17 minutes after launch.
Phase 3: latency, throughput and concurrency
We ran the same tests as for Qwen. The agent workload sends 16 different 4K-token system prompts, as an agent framework does with its instructions and tool schemas, each with a 128-token question; chat sends 1K new tokens; long sends 8K. Every answer is 256 tokens, with thinking off.
TestOutput tok/sTTFT p50TTFT p99ITL p50ITL p99Agent, 1 user104239 ms262 ms8.7 ms9.3 msAgent, 16908477 ms732 ms15.5 ms41.6 msAgent, 641,9351,103 ms7,885 ms21.2 ms49.0 msAgent, 64 (400 requests)2,062684 ms7,207 ms21.5 ms50.0 msChat, 641,4832,662 ms10,563 ms22.6 ms46.1 msLong (8K in), 164415,307 ms5,452 ms15.3 ms40.7 msCold start, 166631,991 ms2,617 ms15.5 ms35.7 msPoisson, 4 req/s875565 ms1,866 ms21.4 ms248 ms
No request failed in any test. What the numbers say:
- One user is fast. 8.7 ms per token is about 115 tokens/s of reading speed, and 239 ms to the first token with a cached 4K system prompt. That is on par with the 8.4 ms we measured for the 27B Qwen on one H200, for a model 100 times larger.
- Throughput scales almost linearly to 16. 8.7× the throughput at 16 requests, with per-token time going from 8.7 to 15.5 ms.
- The prefix cache carries the agent workload. Agent requests at 64 get their first token in 1.1 s; chat requests at 64, which share nothing, wait 2.7 s.
- Long prompts cost prefill. 16 requests × 8K tokens (131K prompt tokens) take about 5.3 s to reach the first token, roughly 25K prompt tokens per second for the node.
The 53-request cap in action
The tail curves make the cap visible. The time to first token is under a second for 83% of requests and about 7 seconds for the slowest 17%, and 17% is almost exactly 11 of 64, the requests that do not fit under the cap and wait for someone else to finish. The per-token tail has a different cause: its slowest 0.2% of tokens take about 230 ms, and with Poisson arrivals the p99 is 248 ms. Our reading, which we did not instrument directly, is that these are decode steps sharing the GPU with a 16K-token prefill chunk; a smaller --chunked-prefill-size would trade prompt throughput for smoother streaming.
Phase 4: inside the kernels
Before starting the server we ran microbenchmarks on one GPU, at Kimi K3’s exact shapes, timed inside CUDA graphs the way the server runs them. (Timed without graphs, FlashInfer’s FP4 GEMM showed ~230 µs of per-call host overhead that completely hid the kernel; measuring the way production runs matters.)
MLA attention, paging and DCP
- DCP pays for itself. Under TP8 each GPU reads a full copy of every request’s latent cache with 12 heads; under DCP8 it reads 1/8 of the cache with all 96 heads. At long contexts that is the difference between 120 µs and 21 µs per layer, before the small cost of merging the partial results.
- An FP8 cache halves TP8 decode time (40.6 → 18.6 µs at 16 × 8K) because TP8 is bandwidth-bound; under DCP8 the gain is smaller (12.9 → 11.5 µs) because each GPU already reads 8× less.
- Paging is free. Scattering the 64-token pages across the pool, as a busy server does, changed decode time by less than 3% in every case (for example 40.9 vs 40.6 µs at 16 × 8K).
Number formats on Blackwell
For prefill-sized batches, Blackwell’s FP4 tensor cores deliver: 5.3 PFLOPS for MXFP4 against 1.7 PFLOPS for BF16 on one expert matrix. At decode-sized batches one expert matrix is only 22 MB and finishes in 4–8 µs in any format, so the 4-bit advantage shows up through the fused MoE kernel, which processes all 16 experts of a layer at once, rather than in a single GEMM. At prefill sizes MXFP4 and NVFP4 performed within about 10% of each other (34 vs 37 µs at 4,096 tokens); NVFP4’s finer scaling (an FP8 scale per 16 values instead of a power of two per 32) costs little speed.
First use costs. The first call of a new kernel variant took up to 3 seconds (MLA under DCP) or 9.5 seconds (the first BF16 GEMM, loading cuBLAS), and the MoE autotune took almost 4 minutes at server start. A production image should ship its JIT and autotune caches.
Cost
- At 64 concurrent agent sessions the node produces output tokens at about $7.65–8.15 per million, half of the Kimi API’s list price of $15 per million output tokens, while keeping every prompt and response on hardware you control.
- For one user the same node costs $152 per million output tokens: a 2.8T model is not the economical choice for a single developer, and the hosted API is.
- The project cost about $49 in metered compute, of which $30 was free credit. That includes downloading 1.56 TB of weights twice, two runs lost to a preemption and a slow loader, and the 40-minute profiling run itself.
Profiling on preemptible GPUs
Our first attempt was preempted six minutes into loading the weights, and Modal restarted it from the beginning. So we split the run into 14 pieces of at most five minutes (three kernel benchmarks, ten serving tests and an export), each uploaded to S3 with a “done” marker the moment it finished, plus a recover_and_check_status step that runs at every start, restores results from S3 and continues at the next unfinished piece. A total time budget and an attempt limit also live in storage, so that restarts cannot run up the bill. The final run finished 13 of the 14 pieces before our account hit its spending limit; only a 256-request burst test was lost, and nothing measured was.
What we would change next time
- Raise the concurrency cap. Spend the 35 GB per GPU that sat idle on state slots, use 4 slots per request instead of 5 (
extra_buffer_lazy), and try a BF16 state. Any of these should lift the 53-request cap and remove the 7-second first-token tail. - Try the FP8 KV cache. It halves MLA decode time under TP8 and would double the KV pool, though under DCP8 the kernel gain is smaller.
- Tune chunked prefill for smoother token streaming when long prompts arrive.
- Ship warm caches (JIT kernels and the 4-minute MoE autotune) in the image, and load weights with prefetch.
Reproduce it
Everything is in our repository under aees/serving: the Modal launcher, the server script, the 5-minute piece runner with S3 recovery, the benchmark scripts and the exported results.
modal run modal_kimi_k3.py::download # 1.56 TB of weights into a Modal volume, CPU only (~12-24 min)
modal run --detach modal_kimi_k3.py::run # 8x B300: kernels, server, serving tests, export
python pieces_kimi_k3.py status --run-id <id> # which pieces are done, read from S3
modal run modal_kimi_k3.py::teardown --confirm # back up results to S3, delete the volumesWith love from the Geodesic team.
Sources
- Moonshot AI, Kimi K3 model card and checkpoint (architecture, benchmarks, MXFP4 quantization-aware training; revision f831ab66).
- Artificial Analysis, Kimi K3 weights release and Intelligence Index, launch evaluation and model page.
- Nathan Lambert, Kimi K3: the open-weights escalation (Interconnects).
- SGLang, Kimi-K3 cookbook (verified B300 recipes).
- FlashInfer, allreduce_fusion and communication API.
- NVIDIA HGX B300 specifications, as summarized in NVIDIA’s HGX AI Factory reference architecture.
- Kimi-K3 on 16× H200: community deployment report.
- Modal, pricing and preemption.
- Our results: the run’s exported data and scripts in the Geodesic repository (
aees/serving/bench_results/kimi-k3_b300_kimi-20261103-145854).
![Deep Reinforcement Learning from Scratch (ML & Infra) [#1]: DQN](https://webproxy.poorya-velaei-d67.workers.dev/https://miro.medium.com/v2/resize:fit:1358/format:webp/1*HThoVe2ngD6eeY_me0qhjw.png)
