Google and Meta bring native TPU training to TorchTitan, supporting Llama 3 and Qwen 3 pretraining
rseroter · x · 2026-10-22
Google and Meta announced TorchTPU integration into TorchTitan, dramatically lowering the barrier to large-scale PyTorch training on Google Cloud TPUs.
- TorchTPU adds a native backend (torch.device("tpu")) inside PyTorch's core interface, requiring only minimal device-init changes
- Previously teams had to rewrite codebases in JAX or rely on legacy torchxla; now Google and Meta co-developed TPU-native execution passes inside TorchTitan
- The fork (github.com/google-pytorch/torchtitan) supports Llama 3, Qwen 3, DeepSeek V3, GPT-OSS, and FLUX.1 pre/post-training, with FSDP2, SimpleFSDP, fused cross-entropy, and Pallas kernels
- The architecture diagram details whole-step XLA compilation, CPU activation offload, and compiled optimizers
More from coding & agent
- Pi Coding Agent Gets a Full Rust Port via Claude, Under 90MB Memory — jasonkneen · 2026-10-22
- Free book 'LangGraph in Small Pieces' teaches agents from blocks to production — truckkicker · 2026-10-22
- What Everyone Is Getting Wrong About Context, per a Founder's New Article — shensi · 2026-10-22
- GitHub for PMs: Version Control Your CLAUDE.md, Skills, and Eval Criteria — aakashgupta · 2026-10-22
- Cloudflare's AI security-audit skill tops GitHub with 38.5k stars — alex_verem · 2026-10-22
- PMs are putting CLAUDE.md, skills and evals in GitHub: 3 repos, 4 workflows only git enables — aakashgupta · 2026-10-22