PyTorch unveils TorchTPU backend so vLLM and SGLang run natively on TPUs
PyTorch · x · 2026-10-02
PyTorch announced vLLM and SGLang engines built on a new PyTorch-native TPU backend, TorchTPU. At PyTorch Conference North America in San Jose, Qi Zhou (Google), Colin Taylor and Angela Yi (Meta) will explain how they made the existing serving stacks work natively on TPUs while preserving schedulers, batching, OpenAI-compatible APIs, and torch.compile workflows familiar to GPU users.
- Engines stay upstream, so new models and features run on TPU without separate TPU-specific implementations
- Production lessons to be shared: compile time, device placement under tracing, KV cache layout, and cross-chip collectives
Related event: PyTorch Unveils TorchTPU, Enabling Native vLLM and SGLang on TPU(2 posts)→
More from Infra
- Mobile AI Factory: agents auto-build on-device SLMs with 19x less GPU time — pmttyji · 2026-11-16
- Stratechery's Fictional Obituary: Imagining Jensen Huang's 2026 Exit From Nvidia — Gregory_C_Allen · 2026-11-16
- vLLM serves DeepSeek V4.1-Flash at 954K tokens/s per GPU, $38.96/GPU-hour on GB300 NVL72 — bookwormengr · 2026-11-16
- Colossus 2's 1.2M B200s could generate ~$400B/year in tokens at DeepSeek Flash rates — bookwormengr · 2026-11-16
- BIS issues rule blocking remote access to US AI chips via third-country data centers — teortaxesTex · 2026-11-16
- Qdrant 1.20.1: 50% fewer cold-storage reads, ~2x faster quantization, agent-ready setup — qdrant_engine · 2026-11-16