Even at temperature 0, 45 of 1,273 LLM answers changed between runs
MaziyarPanahi · x · 2026-10-25
Maziyar Panahi shares methodology details behind an eval run: temperature 0, reasoning on, and forcing each answer to end with "Answer: X" for parsing.
Even so, 45 of the 1,273 answers changed between runs, so the setup runs 3 times and takes the median, with the score ranging from 94.1% to 95.0%. Serving was handled by Baseten at roughly 2.8 seconds per question.
A concrete reminder that temperature 0 doesn't guarantee deterministic LLM outputs, and rigorous eval reports need to disclose run counts and aggregation methods.
More from Models
- Google searches, LLMs predict: an ex-Meta AI engineer explains why the two work differently — HarperSCarroll · 2026-10-26
- Dev builds Nemotron Bolt gatekeeper to block harmful agent tool calls, weighs abliterated models — gnukeith · 2026-10-26
- Top-b sampling: entropy-adaptive decoding beats Top-p on GPQA and cuts variance 10x — cneuralnetwork · 2026-10-26
- Anthropic token volume on OpenRouter up 3.6x since Opus 5.5 launch — petergyang · 2026-10-26
- Claude runs 53 hours unsupervised, beats human designs on 7 of 8 quantum problems — FinanceYF5 · 2026-10-26
- DesignArena data: Gemini 3.8 Flash quality degrades sharply weeks after launch — legit_api · 2026-10-26