vLLM's agent-assisted Kimi K3 optimization hits 842 tokens/s on 16 GB300 GPUs
lmoroney · x · 2026-10-20
The vLLM team detailed using agent-assisted development to build a unified CuTe DSL decode path for Kimi K3 on NVIDIA GB300, covering attention, MoE, and communication across 8/16 GPUs. Results: 842 tokens/s at batch 1 on 16 GB300s with DSpark speculative decoding (synthetic AL=6), 1.5–2× decode throughput gains in tested configs, and up to 1.8× P90 interactivity on AgentX. Key insight: agent-optimized inference code is outpacing human review workflows, so they propose a fast-track RFC model—isolating changes within model code, bounding hardware/workload scope, and designing software for automatic verification of reliability and performance. A landmark case of coding agents writing the hardest GPU kernels, plus an engineering framework for reviewing agent-generated code.
Related event: vLLM Team Tests Coding Agents Writing GPU Kernels for Kimi K3(2 posts)→
More from coding & agent
- Notion AI ships major upgrade: smarter Agents, Auto 90% cheaper, BYOK support — ivanhzhao · 2026-10-20
- iPhone Action Button + Grok voice + HeyGen MCP turns a sentence into an avatar video — HeyGen · 2026-10-20
- Vibe-coded game League of Spawn adds Arenas: 15-minute MOBA team fights — majidmanzarpour · 2026-10-20
- esp32sim-mcp: MCP server lets AI agents build and test ESP32 firmware in a local emulator — EmanueleStrazzullo · 2026-10-20
- GameSummoner one-line local install: prompt-to-game pipeline runs fully local on a 5090 — demonicpigg · 2026-10-20
- Heavy Grok Bot user argues personal agents must stay open-source, shares backup SOP — WolframRvnwlf · 2026-10-20