Philox RNG-to-tensor assignment is extremely sensitive to hardware details

ezyang · x · 2026-11-17

Edward Yang explains why he initially misjudged the distributed RNG problem: "just advance the RNG stream appropriately for each rank" turns out to be wrong, because the exact assignment of Philox to tensors is extremely sensitive to precise hardware details.

That sensitivity is the core reason this reproducibility problem went unsolved for years, per this thread on PyTorch's new stateless RNG API.

Related event: Why TP=1 and TP=2 Initialize Weights Differently: The RNG Problem(2 posts)→

Original post →

More from Infra

Infra channel →