First TPU inference megakernel open-sourced: Kimi K3 hits 709 tok/s, beating GB200 baseline
woosuk_k · x · 2026-09-24
- The inferact team released the first TPU inference megakernel, open-sourced today: the entire Kimi K3 model runs in a single Pallas kernel, reaching 709 tokens/s on low-concurrency decode with DSpark speculative decoding vs 450 tokens/s on their GB200 baseline.
- Without spec decoding, it is roughly 1.4-2x the GB200 baseline at batch sizes 1 through 8.
- Why it works on TPU (explained by woosukk via gaunernst): TPUs have only 1-2 core compute units instead of hundreds of SMs on NVIDIA GPUs — cross-SM synchronization is what kills megakernels on NVIDIA, making them natural on TPU.
Related event: First TPU Inference Megakernel Open-Sourced, Beats GB200(3 posts)→
More from Infra
- Omarchy 4.6 lets you share local AI models across your tailnet, says DHH — alexcovo_eth · 2026-11-12
- Eric Schmidt's foundation funds French NGO fighting against data centers — StephanSturges · 2026-11-12
- Epoch: Open-Weight GLM 5.3 Hacks Six Firefox Versions for ~$1,000 Each; Claude Fable 5.2 Sets ECI Record — Epoch AI · 2026-11-12
- Polymarket prices 28% odds of a state data center moratorium; Torus raises $500M — Polymarket · 2026-11-12
- AWS-wrapper cloud startup shuts down just one month after seed round — thedealdirector · 2026-11-12
- Utah startup Torus raises $500M for shipping container-sized AI data centers deployable in 4 months — Polymarket · 2026-11-12