Philox RNG-to-tensor assignment is extremely sensitive to hardware details
ezyang · x · 2026-11-17
Edward Yang explains why he initially misjudged the distributed RNG problem: "just advance the RNG stream appropriately for each rank" turns out to be wrong, because the exact assignment of Philox to tensors is extremely sensitive to precise hardware details.
That sensitivity is the core reason this reproducibility problem went unsolved for years, per this thread on PyTorch's new stateless RNG API.
Related event: Why TP=1 and TP=2 Initialize Weights Differently: The RNG Problem(2 posts)→
More from Infra
- Why tensor parallelism breaks RNG reproducibility in model init — ezyang · 2026-11-17
- Bittensor's Engy tops Venice's AI inference volume despite 28x smaller valuation — markjeffrey · 2026-11-17
- Fractile raises $720M for AI inference chips, betting inference cost breaks AI economics first — YvesMulkers · 2026-11-17
- DeepSeek V4.1 Flash on Vera Rubin NVL72: 63-93x per-chip throughput vs GB300 in vLLM benchmarks — vllm_project · 2026-11-17
- First DeepSeek V4.1 Flash results on Vera Rubin NVL72: up to 93.3x per-chip throughput vs GB300 — vllm_project · 2026-11-17
- Put an open-source AI on your phone in 10 minutes: free, offline, no account — alex_verem · 2026-11-17