Open-sourced TPU megakernel runs Kimi K3 at 709 tokens/s, beating GB200
vllm_project · x · 2026-09-24
The inferact team open-sourced what they believe is the first TPU inference megakernel, reaching 709 tokens/s on low-concurrency decode for Kimi K3, versus 450 tokens/s on their GB200 baseline (both with DSpark speculative decoding). All 92 of K3's MoE layers run in a single Pallas kernel, with weight prefetching across layer boundaries so transfers for one layer overlap with computation in the previous one. Without speculative decoding it is roughly 1.4-2x the GB200 baseline at batch sizes 1 through 8. The vLLM project amplified the release.
Related event: First TPU Inference Megakernel Open-Sourced, Beats GB200(3 posts)→
More from Infra
- Leaked memo: Meta infrastructure head Santosh Janardhan to leave in early 2027 — dinabass · 2026-11-12
- Tool-call failure rates jump 14x across providers, charts pinpoint inference stack updates — remilouf · 2026-11-12
- Omarchy 4.6 lets you share local AI models across your tailnet, says DHH — alexcovo_eth · 2026-11-12
- Eric Schmidt's foundation funds French NGO fighting against data centers — StephanSturges · 2026-11-12
- Epoch: Open-Weight GLM 5.3 Hacks Six Firefox Versions for ~$1,000 Each; Claude Fable 5.2 Sets ECI Record — Epoch AI · 2026-11-12
- Polymarket prices 28% odds of a state data center moratorium; Torus raises $500M — Polymarket · 2026-11-12