Why temperature 0 isn't deterministic: 1,000 runs of Qwen3-235B yield 80 outputs
JensHonack · x · 2026-10-27
Avi Chawla's technical explainer tackles a popular LLM interview question: why inference isn't deterministic even at temperature 0.
Key points:
- At temperature 0, engines use greedy decoding — picking the highest-logit token is deterministic in itself
- The nondeterminism comes from computing the logits: Thinking Machines sent the same prompt to Qwen3-235B 1,000 times at temperature 0 and got 80 unique completions
- Root cause: limited GPU floating-point precision (fp16 keeps only 11 significant bits) means the order of addition changes results slightly, occasionally flipping the argmax
The post also references Chawla's article on KV, Prefix, Prompt, and Semantic caching in LLMs, covering where input tokens get recomputed and what to do about it.
More from Models
- repligate: noisy context is genuinely 'hard' for LLMs, beyond metaphors — repligate · 2026-10-27
- The trust paradox: only the least trustworthy LLMs will do the sketchiest things — AaronBergman18 · 2026-10-27
- Integrity and refusal may correlate: the fundamental tradeoff behind password refusals — AaronBergman18 · 2026-10-27
- Claude's refusal to type one-time passwords is a real security impediment — AaronBergman18 · 2026-10-27
- repligate argues context full of noise is genuinely 'hard' for LLMs, mulls an info-theory model of breaks — repligate · 2026-10-27
- Fork builds engine to run Qwen at ~400 tok/s on three used V100s — jfinkpottery · 2026-10-27