Experts debate why CUDA atomic ops show fingerprintable nondeterminism across drivers
mike64_t · x · 2026-11-01
A hard-core technical debate between GPU experts on why atomics in the same cubin behave differently across NVIDIA driver versions.
mike64t's position:
- CUDA fat binaries statically ship SM machine code per target arch; instruction scheduling is baked into the cubin, no final recompilation happens
- Variable-latency instructions like atomic add are variable at the ISA level anyway; the puzzle is why the nondeterminism follows a fingerprintable pattern
matiasgoldberg's rebuttal:
- The PTX contract guarantees nothing: NVIDIA not breaking things ≠ guaranteeing stability; drivers can rewrite cubins
- Cycle counts per instruction are unspecified, and atomics depend on timing — all bets are off (analogous to Spectre on x86)
- 600 vs 599 instructions in a fat blob can already flip an atomic op outcome
Useful reading for anyone reasoning about GPU result reproducibility and driver-level effects on compute output.
Related event: GPU Experts Debate Whether Driver Versions Break CUDA Atomic Determinism(9 posts)→
More from Infra
- Is llama.cpp the new Ollama? Reddit debates the fragmented local LLM engine scene — fintip · 2026-11-01
- Researcher teases talk synthesizing a decade of large-scale training systems — _arohan_ · 2026-11-01
- vLLM cold starts on RunPod cut to 14s: DMTCP snapshots where CRIU fails — kbesplv · 2026-11-01
- Meme: writing an LLVM backend for a fully undocumented GPU architecture to ditch closed-source nvcc — mike64_t · 2026-11-01
- Enterprises are distilling domain models, cutting costs from dollars to cents with no accuracy loss — himanshustwts · 2026-11-01
- Prediction: Local Models Will Be the Biggest AI Story of 2027, Disrupting Inference Business — ZeYanjie · 2026-11-01