vLLM brings day-0 support to Moonshot’s Kimi K3

vLLM says Moonshot AI’s Kimi K3 can be deployed on its stack on day 0 once the weights are public, and has published a production-focused deployment guide plus a technical deep dive. The focus is not just that K3 is a 2.8T-parameter MoE model with 1M context and native vision, but how to serve it efficiently: vLLM argues caching is the main systems challenge, and reports 370 tok/s single-stream throughput with speculative decoding. This matters because it shows an open serving stack adapting quickly to ultra-large MoE models with architecture-specific optimizations.

Confirmed

• vLLM describes Kimi K3 as a 2.8T-parameter MoE model that activates 16/896 experts per token, supports 1M-token context, includes native visual capability, and uses Kimi Delta Attention (KDA).
• According to vLLM, the harder problem is not the raw 2.8T parameter count but how to cache the model efficiently during serving. Many K3 layers use fixed-size KDA state instead of a token-growing KV cache, which avoids the usual per-token KV hashing growth and reduces cache pressure.
• For lower latency without accuracy loss, Inferact trained and open-sourced Kimi-K3-DSpark, an MLA-native draft model designed for vLLM. vLLM says its training data reused target hidden states extracted by vLLM itself, enabling cache reuse, and that speculative decoding with this draft model reaches 370 tok/s for single-stream throughput.
• vLLM also lists serving options available for K3 in its stack, including P/D disaggregation, prefix caching, KV offload, and topology-aware MoE backend choices such as megamoe for EP and trtllm when TP>1.

Why it matters

Serving a 2.8T MoE model with low latency is primarily a systems-engineering problem, not just a model-size problem. vLLM’s KDA-state caching approach and the cache-reusing DSpark speculator provide a concrete open engineering path for deploying future ultra-large MoE models.

2026-07-27 ~ 2026-07-29 · 11 related posts

Full story(9 episodes)→

• Episode 1: vLLM brings day-0 support to Moonshot’s Kimi K3(2026-07-27, 11 posts)
• Episode 2: Kimi K3 Now Available for Inference and Fine-Tuning on Fireworks(2026-07-28, 2 posts)
• Episode 3: Kimi K3 2.8T-Parameter Model Runs on 80 RTX 5090s with Zero HBM(2026-07-28, 8 posts)
• Episode 4: Kimi K3 Open-Weight Release Sparks Debate on Open Source and Infrastructure(2026-07-28, 5 posts)
• Episode 5: Kimi K3 Self-Hosting Can Break Even in Under 100 Days(2026-07-28, 4 posts)
• Episode 6: Tinkering with Local Quantized K3 Inference on Mac Hardware(2026-07-29, 2 posts)
• Episode 7: Kimi K3 Open Weights Demand Data Center Hardware(2026-07-29, 3 posts)
• Episode 8: vLLM Hits 464 tok/s on Kimi K3 with 4 GB300 Systems(2026-07-29, 2 posts)
• Episode 9: Kimi K3 Gets Day-0 vLLM and AMD Support Across Clouds(2026-07-30, 14 posts)

• vLLM says K3’s 2.8T parameters were easy; caching them was the hard part — vllm_project · 2026-07-27
• vLLM adds day-zero support for Kimi K3, a 2.8T MoE model with 1M context — AAAzzam · 2026-07-27
• [source] Kimi K3 uses fixed-size KDA state instead of a growing KV cache — vllm_project · 2026-07-27
• vLLM lists Kimi K3 serving options with disaggregation, KV offload and MoE backends — vllm_project · 2026-07-27
• Inferact’s Kimi-K3-DSpark draft model reuses MLA caches to speed up vLLM serving — vllm_project · 2026-07-27
• [source] Open-sourced Kimi K3 speculator lifts single-stream throughput from 118 to 370 tok/s — vllm_project · 2026-07-27
• Kimi K3 with 2.8T parameters and 1M context now supported on vLLM — ricklamers · 2026-07-27
• vLLM ships day-0 serving support for Moonshot’s 2.8T-parameter Kimi K3 — vllm_project · 2026-07-28
• vLLM ships a Day-0 deployment guide for Kimi K3, a 2.8T MoE with 1M context — AccBalanced · 2026-07-28
• [source] vLLM posts day-zero Kimi K3 support with 2.8T MoE serving details — vllm_project · 2026-07-29

1 near-duplicate retellings: woosuk_k