Zephon x TorchTitan: bit-identical global batches when resuming training on different GPU counts
PyTorch · x · 2026-10-17
PyTorch's blog details Zephon's TorchTitan integration: resuming a 1B-model run from a 16-GPU checkpoint on 8, 32, and 64 GPUs produced SHA-256-identical global batches. Zephon tokenizes, packs, shuffles, and mixes data inside the training job, avoiding preprocessed corpus copies. Constraints: data, tokenizer, seed, mixture, context length and global batch size must stay unchanged; lanes must divide evenly across replicas. Losses can still differ due to floating-point reduction order. Benchmarks show 22.4k-29.8k usable tokens/sec per GPU vs 1.1k-9.2k for internal data pipelines.
More from Infra
- Nebius CEO meets Dutch PM as commentator calls it 'the Google of Europe' — robleclerc · 2026-10-17
- SGLang powers Reflection AI's new Beam model in production with day-0 support — ying11231 · 2026-10-17
- speech.cpp v0.8.8 adds Fish Audio drama-3 and M5 Mac speedups, no Python/CUDA needed — solyarisoftware · 2026-10-17
- Joke read: Altman praising the Roadster hints OpenAI may need to rent GWs from SpaceX — matt_slotnick · 2026-10-17
- NVIDIA releases LiLiCorr speculative decoding draft for Nemotron 3.5 Bolt to speed up 74B serving — _akhaliq · 2026-10-17
- Multi-GPU MTP can hurt decode speed: 82 vs 70 tok/s on 4x RTX 3090 — macboller · 2026-10-17