Hackathon achieves fully deterministic LLM inference across devices with no quality or speed loss
teortaxesTex · x · 2026-08-12
At the SpaceX AI hackathon, developer nathanrs demonstrated how to make LLM inference fully deterministic across devices without sacrificing quality or speed. Using Qwen3-0.6B, they produced identical hashed logits for a 512-token generation across GPUs (A100, H100, Apple M5 Max) and CPUs (AMD EPYC, Intel Xeon).
The key issue is that floating-point addition is not associative, so accumulation order varies with hardware and kernel, causing different outputs even at temperature 0. Integer arithmetic is associative, but quantized models dequantize non-linear ops (softmax, normalization, SiLU) to float and requantize, introducing variability. The solution modifies these ops to run in integer domain, achieving full determinism.
More from coding & agent
- Hidden Hermes Agent Commands: Automate Workflow Learning and Context Compression — Teknium · 2026-08-12
- Waku: An Open-Source Native Desktop App for Coding Agents Built with Rust+GPUI — dotey · 2026-08-12
- Open-Source Claude Artifacts Alternative 'Llama Coder' Generates Apps with One Prompt — tom_doerr · 2026-08-12
- GitHub Trending: Semantica – Graph-Native Infrastructure for AI Systems — adnan_hashmi · 2026-08-12
- AI Coding Tip: Have LLMs Generate TESTING.md Instead of Reading Code Line by Line — ykdojo · 2026-08-12
- Parallel AI Coding Without Worktree: Handling Conflicts in Agents — dotey · 2026-08-12