vLLM says K3’s 2.8T parameters were easy; caching them was the hard part
vllm_project · x · 2026-07-27
vLLM explains that fitting K3’s 2.8T parameters was not the hard part; caching them was.
Because most K3 layers use fixed-size KDA state instead of a growing KV cache, there is no per-token KV to hash. Prefix caching had to be redesigned around that architecture, and the post says every hybrid model after K3 inherits the result.
The attached diagram also shows how K3 combines components such as Stable LatentMoE, Gated MLA, and KDA inside a multi-block stack.
More from Infra
- Baseten adds Kimi K3 to its Model APIs on launch day — baseten · 2026-07-27
- MoonshotAI open-sources MoonEP for perfectly balanced expert parallelism — teortaxesTex · 2026-07-27
- Moonshot releases Kimi K3, a 2.8T MoE model with 1M context and 423 tok/s serving — ricklamers · 2026-07-27
- How to run a private local AI on many 16GB laptops with Ollama and Gemma 4 — minchoi · 2026-07-27
- Vercel adds Kimi K3 on US providers with zero-data-retention support — cramforce · 2026-07-27
- Kimi K3 is said to cost more than 2× as much to serve as V4 — teortaxesTex · 2026-07-27