Zephon's TorchTitan integration resumes training across GPU counts with identical batches
josh_wills · x · 2026-10-17
PyTorch's latest blog details Zephon's TorchTitan integration, which preserves identical global batches when training resumes on a different GPU count. In a 1B-parameter model test, training resumed from a 16-GPU checkpoint on 8, 32, and 64 GPUs, and the SHA-256 hash of every global batch matched the original run.
- Zephon tokenizes, packs, shuffles, and mixes data inside the training job, avoiding a preprocessed corpus copy.
- Data, tokenizer, seed, mixture, context length, and global batch size must stay unchanged when resuming; lane counts must divide evenly across data-parallel replicas and per-step batches across lanes.
- Training losses can still differ due to floating-point non-determinism.
More from Infra
- Broadcom's Ultra and Jericho Are the Underrated Half of the AI Ethernet Story — BenBajarin · 2026-10-17
- Analyst rebuts 'China floods HBM by 2028' claim, citing 5-7 year DRAM gap — JOBhakdi · 2026-10-17
- Bitter lesson for local LLMs: below 100 TPS, you won't actually use it — TheZachMueller · 2026-10-17
- AI is pulling quantum timelines forward, but hardware is at the vacuum-tube stage — robleclerc · 2026-10-17
- SentryWard: 32 of 85 sampled GPU kernels harbor race and sync bugs, formal verification for AI-generated code — JiaZhihao · 2026-10-17
- llama.cpp merges IBM Granite switch support in PR #30676 — jacek2023 · 2026-10-17