Debugging exact DDP checkpoint resumption: CUDA atomics, Triton autotune and bucket layout pitfalls

ReinforcedKnowledge · reddit · 2026-08-28

The author tested exact checkpoint resumption in DDP training (Qwen3.5-0.8B, open-Instruct, 4×A100): a continuous 10-step run vs one checkpointed at step 5 and resumed. Even after verifying every state — optimizer, model, schedulers, Python/NumPy/Torch CPU RNG, every CUDA RNG stream, data packs and document maps — the runs still diverged.

Hooks everywhere revealed two pre-communication culprits:

The default causal-conv1d backward accumulated weight/bias gradients with unordered CUDA atomics;
Triton autotune decisions were process-local and interacted with a shared disk cache, so resumed runs could pick different kernels. Making the conv backward deterministic and freezing rank-private autotune records fixed most of it.
One subtle source remained: 261 of 320 post-reduce gradients differed because the DDP reducer bucket layout differed (61 buckets for the continuous run vs 1 bucket for the fresh resumed wrapper) — the gradient-ready order history and rebuild state are simply not part of training checkpoints.

The final fix: initialize the fresh reducer with a deterministic synthetic full-graph backward, explicitly rebuild buckets, clear gradients, and restore protected state before real data arrives. Useful only when strict reproducibility actually matters.

Original post →

Debugging exact DDP checkpoint resumption: CUDA atomics, Triton autotune and bucket layout pitfalls

More from Infra