vLLM Team's Inferact Runs Kimi K3 57% Faster on 16 TPUs Than GB200 With Open-Sourced Megakernel
量子位 · wechat · 2026-09-26
Inferact, founded by the original vLLM team ($150M seed at $800M valuation), open-sourced a megakernel that runs Kimi K3 at 709 tokens/s on 16 TPUv7 — 57% faster than 16 GB200 at 452 tokens/s, with zero accuracy loss. The trick: fusing all 92 MoE layers into one Pallas program, eliminating kernel boundaries and enabling cross-layer weight prefetch. TPU's software-managed 64MiB VMEM beats GB200's hardware-scheduled 38MiB despite lower HBM bandwidth, showing the gap is software, not silicon.
More from Infra
- Running Qwen2.5-Coder-3B on Android: Jupyter + Pydantic Evals on-device eval pipeline — alex_b3b · 2026-11-11
- 22% of Nvidia revenue now books to Singapore, but only ~1% is local buyers — Gregory_C_Allen · 2026-11-11
- Huawei's Kirin 9050 NPU claims offline 30B MoE on smartphones, +140% perf — AIFlow_ML · 2026-11-11
- Stratechery: Intel vs AMD's diverging earnings and Intel's need for an old-new path — Stratechery · 2026-11-11
- a16z Podcast: Mike Volpi on What the 1990s Internet Boom Teaches Us About the AI Buildout — a16z Podcast · 2026-11-11
- Alibaba files to list chip-design unit T-Head in Hong Kong to fund AI push — pstAsiatech · 2026-11-11