PyTorch megakernel runs 816B MoE on one TPU v7 host at 433 tok/s, beats GB200 on cost
PyTorch · x · 2026-11-11
PyTorch highlighted community inference work on Muse Spark Open: all 62 layers of the 816B MoE run as one Pallas program on a single TPU v7 host, via SGLang on the TorchTPU PyTorch backend.
Key numbers:
- 433 tok/s per user; up to 25.9K tok/s per chip on AgentX (1M context)
- 1.94x faster per token than the previous path
- DFlash2 speculative decoding adds 2.0x per token
- fp8 KV, HiCache, and session-aware eviction for long agent sessions
The accompanying chart shows TPU v7 (four chips, $1.21/chip-hr, FP8 compute with NVFP4 experts) beats GB200 FP4 ($1.86/GPU-hr) on million tokens per dollar across the AgentX p90 interactivity curve, with the gap widening at higher interactivity.
More from Infra
- AI infrastructure bet hits 3.6% of GDP, driving nearly 40% of US growth — NinaDSchick · 2026-11-11
- Google Colab bans user for 'abusive content' over NSFW AI image generation — EdgyUsername_0529 · 2026-11-11
- Tendrils raises $30M led by Benchmark to rebuild general-purpose computing — saranormous · 2026-11-11
- Microsoft lobbies Trump admin to keep Chinese hardware in US AI data centers — kevinnbass · 2026-11-11
- WSJ: Data center buildouts are pulling labor and resources from housing and manufacturing — RebeccaBellan · 2026-11-11
- The AI Race Is Now an Electricity and Permitting Race, Not Just a Model Race — TansuYegen · 2026-11-11