Zephon brings elastically deterministic data loading to TorchTitan with identical batches across GPU counts
josh_wills · x · 2026-10-17
PyTorch's blog details Zephon's integration with TorchTitan: a 1B-param model trained on 16 GPUs resumed from step 150 on 8, 32, and 64 GPUs, and the SHA-256 hash of every global batch matched the original run. Zephon tokenizes, packs, shuffles, and mixes data inside the training job, avoiding costly preprocessed corpus copies. Determinism matters more now that agents run experiments in parallel and can't notice when results depend on GPU count rather than the change being tested. Benchmarks show 22.4k-26.6k usable tokens/sec per GPU versus 1.1k-9.2k for standard loaders. Losses can still differ slightly due to floating-point reduction order.
More from Infra
- Nebius CEO meets Dutch PM as commentator calls it 'the Google of Europe' — robleclerc · 2026-10-17
- SGLang powers Reflection AI's new Beam model in production with day-0 support — ying11231 · 2026-10-17
- speech.cpp v0.8.8 adds Fish Audio drama-3 and M5 Mac speedups, no Python/CUDA needed — solyarisoftware · 2026-10-17
- Joke read: Altman praising the Roadster hints OpenAI may need to rent GWs from SpaceX — matt_slotnick · 2026-10-17
- NVIDIA releases LiLiCorr speculative decoding draft for Nemotron 3.5 Bolt to speed up 74B serving — _akhaliq · 2026-10-17
- Multi-GPU MTP can hurt decode speed: 82 vs 70 tok/s on 4x RTX 3090 — macboller · 2026-10-17