vLLM merges NVIDIA's GB300-specialized Kimi K3 decode: 1.5-2x speedup
marksaroufim · x · 2026-10-18
vLLM merged PR #61867, an NVIDIA-developed latency-specialized Kimi K3 decode implementation for GB300 with CuTeDSL persistent kernels and PDL, achieving 1.5-2x decode speedups (e.g. 3.90ms vs 7.38ms per step at B1, TP16, FP8 KV cache). It supports TP8/TP16, NoSpec up to B64, DSpark verification, and DCP=TP, enabled opt-in via a decodebackend config. marksaroufim reflects on how OSS handles large AI-generated PRs — merge into a standalone folder, shifting the problem from composability to change discovery, "a repo becomes a blog with a known audience" — and argues "megakernel" is a misnomer here.
Related event: NVIDIA's vLLM Optimization Boosts Kimi K3 Decoding on GB300(2 posts)→
More from Infra
- Running Qwen 35B at 82 tok/s with a mismatched 5070 Ti + external 3070 build — hereforcheesee · 2026-10-18
- China building ~69 power semiconductor fabs across 58 suppliers — zephyr_z9 · 2026-10-18
- AI data centers need $3.5T yearly revenue to stay solvent, research estimates — TheseFact · 2026-10-18
- SGLang Summit sponsors grow from 11 to 36 in one week; Lip-Bu Tan, Lilian Weng to speak — BanghuaZ · 2026-10-18
- Yacine jokes his datacenter KPI is gigawatt-hours: get paged if not burning enough power — yacineMTB · 2026-10-18
- 1.68x inference speedup drops, directly accelerating GRPO post-training — yacineMTB · 2026-10-18