SemiAnalysis's 1.68x vLLM speedup claim for Kimi K3 questioned over TP16 vs TP8 baseline
JiaZhihao · x · 2026-10-18
SemiAnalysis reported NVIDIA hit 423 tok/s/user on the 2.8T-parameter Kimi K3 via new upstream vLLM optimization — a 12,000-LoC decode-specialized Megakernel by NVIDIA CentML's Benjamin, written in Cute DSL with aggressive kernel fusion, with DeepSeek v4.1 and others in the pipeline. But JiaZhihao flags a methodology concern: the speedup appears to compare a TP16 setup against a TP8 baseline, i.e. 2x the GPU resources — an apples-to-apples TP configuration would be the fair test.
More from Infra
- SlateDB+Lance+Datafusion stack saves customer millions, as AI agents unbundle big data systems — sh_reya · 2026-10-18
- Paying $0.50 per 10 seconds for video gen APIs — when is self-hosting worth it? — Key-Lychee-913 · 2026-10-18
- vLLM and NVIDIA push Kimi K3 decode to 842 tokens/s on GB300 with fused CuTe DSL path — vllm_project · 2026-10-18
- Blackstone's AI empire: $200B data centers, $95B chip financing, bets on Anthropic and OpenAI — SuB8u · 2026-10-18
- Fix NVIDIA RTX VFX extension install failures in ComfyUI with one pip command — Mad4reds · 2026-10-18
- Running an 816B open model on M3 Ultra 256GB: first successful decode at 4 t/s — challis88ocarina · 2026-10-18