GPU Experts Debate Whether Driver Versions Break CUDA Atomic Determinism

NVIDIA GPU engineer matiasgoldberg and graphics/computing expert mike64t held a multi-round, deep technical debate on November 1 over whether GPU driver versions affect device-side code fingerprints and whether CUDA atomics are deterministic. The core question: when the same cubin (compiled GPU machine code) runs under different driver versions, are the results of atomic operations (atomic add) reproducible?

Confirmed

• Both sides reached some consensus on the CUDA compilation model: a fat binary directly packages a cubin for each target sm version, with no final-stage recompilation; "virtual sm" merely marks that an sm100 binary originated from compilation against a virtual architecture, and almost no one in the compute space recompiles PTX (mike64t, matiasgoldberg).
• mike64t clarified: once memory instructions like LDG.E or ATOMG.ADD are issued, timing becomes "unpredictable" from the ISA model's perspective, though the observed latency is not inexplicable.

Points of Contention

• matiasgoldberg argued that the PTX contract does not guarantee stability of compiled output: NVIDIA choosing not to break compatibility is different from a formal guarantee; even though the cubin is fixed, the driver can still decide to rewrite it. Code must be translated into the final ISA, and this translation can change across driver versions; even if the translation stays the same, small changes in the order of hardware invocations (e.g., commands A, B, C becoming C, A, B) can affect read behavior.
• He further noted that GPUs only guarantee the final value of an addition, not whether it completes in 3 cycles or 1, nor which core unit handles it; once an operation depends on atomics—whose correctness depends on timing—all guarantees vanish. His extreme example of "600 vs. 599 instructions" was meant to illustrate the point, not to be taken literally.
• mike64t systematically countered with the CUDA compilation model: fat binaries completely freeze instruction scheduling for each sm version, so there is no driver recompilation that could change fingerprints.

Why It Matters

The debate touches on the fundamental contract of determinism in GPU computing: if driver versions can alter the cubin or execution timing, cross-driver reproducibility of anything relying on atomic operations cannot be guaranteed—with real implications for scientific computing, benchmarking, and result auditing.

2026-11-01 ~ 2026-11-01 · 9 related posts

• Do driver versions change GPU device-code fingerprints? Experts debate compilers vs cubins — matiasgoldberg · 2026-11-01
• Final ISA translation changes per driver, reordering GPU command execution — matiasgoldberg · 2026-11-01
• [source] Nobody recompiles PTX in compute: cubin scheduling is baked into the binary — mike64_t · 2026-11-01
• [source] NVIDIA GPU engineer explains CUDA fat binaries: PTX is a virtual ISA, no final recompilation — matiasgoldberg · 2026-11-01
• GPUs guarantee the value, not the timing — and atomics depend on timing — matiasgoldberg · 2026-11-01
• PTX guarantees nothing: drivers can still rewrite your compiled cubins — matiasgoldberg · 2026-11-01
• 600 instructions or 599: nothing guarantees which, and it flips atomics — matiasgoldberg · 2026-11-01
• [source] Experts debate why CUDA atomic ops show fingerprintable nondeterminism across drivers — mike64_t · 2026-11-01
• Dev deep-dive on CUDA instruction latency: atomic ops are variable-latency at the ISA model level — mike64_t · 2026-11-01