PyTorch's TorchTitan adds DeepSeek-V4, Kimi K3 support, RL training and big scale gains
PyTorch · x · 2026-10-22
The PyTorch team shipped a major TorchTitan update over the past year while keeping its minimal, extensible ethos:
- New SOTA model support including Qwen3-VL, Qwen3.5, Kimi K2.7, Kimi K3, and DeepSeek-V4, all as full training-stack integrations across the 5D parallelism stack
- A composable config system built on typed Python recipes, letting users swap parallelization strategies, model components, or hardware kernels without separate forks
- Optimized kernels from Attention Gym: Gated DeltaNet, Kimi Delta Attention, sparse attention
- Dramatic out-of-the-box performance at scale, plus new RL support (TitanRL) and communication features
One shared model.py now drives pretraining (TorchTitan), RL training (TitanRL), and generation via vLLM inference adapters.
Related event: PyTorch's TorchTitan Gets Major Update With New SOTA Model Support(2 posts)→
More from Infra
- 8x NVIDIA DGX Spark cluster with 200GB switch becomes SF's new flex — beffjezos · 2026-10-22
- NVIDIA Cosmos-3-Nano world models go live on Reactor's robot inference cloud — chris_j_paxton · 2026-10-22
- Abacus.AI CEO beta-tests GPU-hosted apps on SuperComputer, runs 3D flying-car game — bindureddy · 2026-10-22
- 2-bit draft in VRAM + exact residual in RAM: speculative decoding for models that don't fit your GPU — Happy_Guitar3521 · 2026-10-22
- Seroter's daily list: OpenAI models land on Google Cloud, PHP CVEs, Genkit Go 1.14 — rseroter · 2026-10-22
- inference.net tops OpenRouter third-party hosts at 14% share, serving ~10% of daily volume — Ronangmi · 2026-10-22