Why tensor parallelism breaks RNG reproducibility in model init

ezyang · x · 2026-11-17

Edward Yang uses tensor parallelism to illustrate the RNG SM-dependence problem: a model's weight init requires one RNG kernel call at TP=1, but two calls on different nodes at TP=2 — yet results should be identical across both configurations.

He notes he'd long been aware of this parallelism-invariance issue from another setting: ensuring parameter initialization doesn't depend on the choice of parallelism.

Related event: Why tensor parallelism breaks RNG reproducibility in model init(2 posts)→

Original post →

More from Infra

Infra channel →