SGLang-TorchTPU launches with Google and Meta, hits 770 tok/s decode in production
ying11231 · x · 2026-10-22
SGLang launched SGLang-TorchTPU, a TPU plugin built with Google's TPU team and Meta's PyTorch team, already powering production inference at Meta. It brings top open models (Muse Spark Open, GLM-5.3 Flash, Kimi K3, Ling 3.1 Flash, MiniMax H3) to TPUs, SGLang's production features for large agentic workloads, and a custom MegaKernel achieving 770 tok/s decode at bs=1 on InferenceX 8K/1K. Apache-2.0 licensed.
Related event: Meta runs AI inference on Google TPUs in production(2 posts)→
More from Infra
- Ollama Runs Open Models on NVIDIA Blackwell via Crusoe, Live in Two Weeks — ollama · 2026-10-22
- Claim: every 2nd/3rd-tier inference provider now has a recipe for 90%+ compute margins — zephyr_z9 · 2026-10-22
- NVIDIA keynote: silicon photonics CPO switches hit 102.4 Tb/s, Vera Rubin architecture unveiled — thealexbanks · 2026-10-22
- PyTorch Foundation Adds Nine Members, Nebius, Nscale and Shopify Join as Platinum — PyTorch · 2026-10-22
- GPU startup Flappy Airplanes raises $1B Series A co-led by Index and Lux — vitaliychiley · 2026-10-22
- Crusoe unveils Tinker API-compatible LLM post-training platform at PyTorch Conference — PyTorch · 2026-10-22