vLLM and NVIDIA push Kimi K3 decode to 842 tokens/s on GB300 with fused CuTe DSL path
vllm_project · x · 2026-10-18
vLLM and NVIDIA built a unified CuTe DSL decode path for Kimi K3 on GB300 NVL72, fusing attention, MoE and communication. It hits 842 tokens/s at batch 1 with DSpark across 16 GPUs, delivers 1.5–2× higher decode throughput in several configurations, and up to 1.8× higher P90 interactivity on the SemiAnalysis AgentX workload.
The write-up also tackles a workflow problem: agent-assisted kernel optimization outpaces manual review. A proposed fast-track RFC manages risk by isolating changes within model code, bounding supported hardware/workloads, and adding automated verification of reliability and performance.
More from coding & agent
- Nimble Notes: A Local-First Desktop Note App for Vibe-Coding Workflows — Nimblecloud13 · 2026-10-18
- Two Claude Sessions Collab to Code-Render a 16,074-Frame Music Video — Se777enUP · 2026-10-18
- RAG isn't dead: it's still the cost-efficient answer for millions of documents — ashishllm · 2026-10-18
- Day drinking and merging a dozen compiler PRs: AI-coded productivity flex — cramforce · 2026-10-18
- Xcode's type-to-filter menus are built into AppKit — enable them in any app with one defaults command — zats · 2026-10-18
- SciML bench: open-weight models miss all bugs, approve broken code; frontier models shine — ChrisRackauckas · 2026-10-18