SGLang hits 25.9K tok/s per chip on TPU v7, running 816B MoE as one megakernel
ying11231 · x · 2026-11-12
SGLang, RadixArk and Google show Muse Spark Open running on a single TPU v7 host via SGLang-TorchTPU under SemiAnalysis' brutal AgentX workload:
- Megakernel + DFlash2 speculative decoding: 3x faster decode at small batch; 433 tok/s per user and up to 25.9K tok/s per chip at 1M context, with all 62 layers of the 816B MoE running as one Pallas program — 1.94x faster per token than the previous path, 2.0x with DFlash2
- Long context: 18% faster decode at 500K tokens (independent KV streams, max-free attention), fp8 compact KV cache doubles context per chip
- Agent throughput: fused MoE kernel + sequence parallelism speed up huge prompt prefill; HiCache and session-aware eviction add up to 77% throughput
Code, recipes and multi-hardware deployment docs (GB300/B300/B200/H200/MI355X/Xeon) are open source.
Related event: SGLang Runs 816B MoE on a Single TPU v7 Host(2 posts)→
More from coding & agent
- Stanford's new CS329Z course dissects agent architecture and real failure cases — dotey · 2026-11-12
- Dev builds sketch-guessing app with new Gemini Decisions API, guesses terrible drawings — fofrAI · 2026-11-12
- Together Compiler: agent-driven continual compilation for AI inference — togethercompute · 2026-11-12
- Book Review: Building AI Agents for Finance — a practical multi-agent patterns reference — bibryam · 2026-11-12
- Kalshi partners with AnyAlpha to let AI agents trade prediction markets 24/7 — ycombinator · 2026-11-12
- Google solicits Kubernetes users to help shape Agent Substrate for Managed Agents — rakyll · 2026-11-12