Why tensor parallelism breaks RNG reproducibility in model init
ezyang · x · 2026-11-17
Edward Yang uses tensor parallelism to illustrate the RNG SM-dependence problem: a model's weight init requires one RNG kernel call at TP=1, but two calls on different nodes at TP=2 — yet results should be identical across both configurations.
He notes he'd long been aware of this parallelism-invariance issue from another setting: ensuring parameter initialization doesn't depend on the choice of parallelism.
Related event: Why tensor parallelism breaks RNG reproducibility in model init(2 posts)→
More from Infra
- Bittensor's Engy tops Venice's AI inference volume despite 28x smaller valuation — markjeffrey · 2026-11-17
- Fractile raises $720M for AI inference chips, betting inference cost breaks AI economics first — YvesMulkers · 2026-11-17
- DeepSeek V4.1 Flash on Vera Rubin NVL72: 63-93x per-chip throughput vs GB300 in vLLM benchmarks — vllm_project · 2026-11-17
- First DeepSeek V4.1 Flash results on Vera Rubin NVL72: up to 93.3x per-chip throughput vs GB300 — vllm_project · 2026-11-17
- Put an open-source AI on your phone in 10 minutes: free, offline, no account — alex_verem · 2026-11-17
- Cloudflare open-sources Kitesurf browser engine for agents, ships DO and spend-limit updates — ritakozlov · 2026-11-17