Making LLM inference bit-identical across GPUs and CPUs with integer-only arithmetic
CatAstro_Piyush · x · 2026-11-13
Inference isn't deterministic across devices because floating-point addition isn't associative—accumulation order varies with hardware and kernels, so even temperature 0 can yield different outputs. By doing every op in integers (including softmax and norms), integer addition's associativity gives bit-identical results. Demoed with Qwen3-0.6B: identical hashed logits over a 512-token generation across A100, H100, Apple M5 Max, AMD EPYC and Intel Xeon, with no loss to quality or speed. Repo and blog post linked.
More from Infra
- Japan inference startup ai& deploys heterogeneous chips, plans 100MW+ data centers — DavidBennett__ · 2026-11-13
- Accord GPU open beta queues ComfyUI, Blender, Ollama to stop VRAM fights — MrAddams_LibraLogic · 2026-11-13
- 27,262 thinking tokens, zero answer—and a 'resolved' Google Cloud billing case — Plenty-Jackfruit-218 · 2026-11-13
- Ex-Nvidia CUDA engineer: 'CUDA is not the moat — Nvidia's only moat is its people' — AlpinDale · 2026-11-13
- AMD expands PsiQuantum partnership with planned equity investment in photonic quantum computing — sven_ai · 2026-11-13
- Nvidia downgrades hit seven as analysts say depths 'not fully probed' — SumitGup · 2026-11-13