Zephon's TorchTitan integration keeps global batches bit-identical across GPU changes
josh_wills · x · 2026-10-17
DatologyAI's Zephon now integrates with PyTorch's TorchTitan: when you change GPU count mid-run, a typical data loader changes data order and possibly results, while Zephon keeps every global batch identical.
In a 1B-parameter test on the PyTorch blog, training resumed from a 16-GPU checkpoint on 8, 32, and 64 GPUs, and the SHA-256 hash of every global batch matched the original run.
Key details:
- Zephon tokenizes, packs, shuffles, and mixes data inside the training job, avoiding a preprocessed corpus copy
- Data, tokenizer, seed, mixture, context length, and global batch size must stay unchanged on resume
- Lane count must divide evenly across data-parallel replicas, and each optimizer step's batch must split evenly across lanes
- Training losses can still differ due to floating-point reduction order
More from Infra
- Generating 4K 60FPS Video on Just 16GB of VRAM, According to a Reddit Share — LanceCampeau · 2026-10-17
- YC-backed Oway launches Internet of Industrials protocol, first model cuts freight costs ~30% — ycombinator · 2026-10-17
- Visko partners with Google Cloud to scale real-time world generation on TPUs — _vztu · 2026-10-17
- Analyst argues CoreWeave isn't dying, sees Anthropic funding roadshow beating expectations — ivan_bezdomny · 2026-10-17
- AI agents are having their Kubernetes moment: agent logic sinks into the platform layer — bibryam · 2026-10-17
- Why inference providers must choose: fast (Groq) or cheap (CoreWeave), not both — AccBalanced · 2026-10-17