vLLM boosts Kimi K3 serving throughput 2.2-2.8x with scheduler, KDA and MoE kernel optimizations
vllm_project · x · 2026-09-17
The vLLM team published a deep dive on optimizing Kimi K3 serving: on a B300 node with an 8K/1K workload at TP8, throughput improved 2.2–2.8x vs v0.27.1, latency dropped 56–60% and TTFT fell 72–85% across concurrency 1/4/16.
Key changes span the full stack:
- Adaptive speculative-token budgets (DSpark speculation, 8 tokens)
- Internal KDA prefix checkpoints and zero-copy mixed KDA batches
- Deferred MXFP4 expert kernel finalization
- ReplaySSM, prefill/decode disaggregation with cache offload, decode context parallelism
The post notes scheduler limits and small tensor copies mattered as much as large GEMMs, and ships reproducible server configs and benchmarks. Work is tracked in issue #50587.
More from Infra
- Huawei unveils Peerium architecture and Atlas 960E SuperPoD scaling to 1M NPUs — pstAsiatech · 2026-09-18
- Agentic traffic now exceeds 70% of all inference traffic, says SemiAnalysis — NinaDSchick · 2026-09-18
- Qwen Flash breaks the big-memory myth: composable PCs make sense again — sn2006gy · 2026-09-18
- Four rate limiting algorithms explained in one visual guide — _jaydeepkarale · 2026-09-18
- Robotics startup Watney raises $80M to expand data-center-building robot fleet — Polymarket · 2026-09-18
- GLM's blog on building its own inference infra read as a shot at Dario — Elux91 · 2026-09-18