TorchTitan adds 5D parallelism for DeepSeek-V4, Qwen3.5, Kimi K3; Muon optimizer 79% faster
PyTorch · x · 2026-10-22
The PyTorch team shipped major TorchTitan updates:
- New SOTA model support: full 5D-parallelism training-stack integrations for DeepSeek-V4, Qwen3.5/Qwen3-VL, Kimi K2.7/K3, and Muse Spark Open, plus a composable typed-recipe config system and Attention Gym kernels (Gated DeltaNet, Kimi Delta Attention, sparse attention) — no more model forks.
- TitanRL: unified post-training and agentic workloads on the same training engine, sharing model code across pretraining, RL, and vLLM generation.
- Performance: FlexShard cuts distributed Muon optimizer time by up to 79%; torchcheckpointing drops checkpoint blocking to 34 ms on 512 GPUs.
More from Infra
- Thread: speculative decoding speeds up inference without changing weights or sampling — cneuralnetwork · 2026-10-22
- Speculative Decoding Explained: How Draft Models Yield k+1 Tokens Per Pass Without Changing the Distribution — cneuralnetwork · 2026-10-22
- Cloudflare reveals it powers OpenAI's ChatGPT sites with Workers and D1 — ritakozlov · 2026-10-22
- DeepSeek has open-sourced every layer of an AI lab, from weights to inference engine — examachine · 2026-10-22
- Meta's Prometheus: hyperscale GPU clusters run at 87% usable, 13% kept as hot spares — SumitGup · 2026-10-22
- Redditor converts missing models to OpenVINO IR and takes requests — Sudden_Topic5154 · 2026-10-22