SGLang Ecosystem Relay: Kimi K3 Hits 423 tok/s on Day-0 with Deep Optimizations
songhan_mit · x · 2026-07-30
The open-source inference framework SGLang achieved up to 423 tok/s on Kimi K3 on its day-0 release (measured on GSM8K), with native RL support ready.
This was powered by a massive relay race across the open-source ecosystem and tech giants. NVIDIA and AMD contributed serious kernel and hardware enablement, while KVCache pushed support for PD disaggregation and HiCache. Cloud providers like Modal, Baseten, and DigitalOcean engaged in co-development, testing, and compute provisioning.
SGLang deeply optimized K3's new architecture using fused KDA decode kernels, DP attention, DSpark, PD disagg, and KDA-aware prefix caching, enabling production-ready efficiency for the largest open-source model.
More from Infra
- SkyRL ships Day-0 RL support for Beam with +62% LoRA decode throughput in vLLM — vllm_project · 2026-10-16
- DFlash2 delivers up to 3.03x decode-only speedup for Beam across five benchmarks — vllm_project · 2026-10-16
- vLLM's Beam serving kernels cut GB200 forward time from 8.15 to 6.47 ms at TP8 batch 128 — vllm_project · 2026-10-16
- Reflection's 501B MoE Beam hits 463 tok/s on 8x GB200 with NVFP4 at day 0 — zhyncs42 · 2026-10-16
- Nemotron 3.5 Bolt runs 16 concurrent streams on a single DGX Spark — NVIDIAAI · 2026-10-16
- Why your computer can't store 0.2: an IEEE 754 primer — Franc0Fernand0 · 2026-10-16