Your LLM Evaluation Is Lying to You: Even at Temperature 0, 1000 Runs Yield 80 Answers

On October 30, Irem Ergun published a long-form tweet thread titled "Your LLM Evaluation Is Lying to You," adapted from her talk of the same name at AI Connect Istanbul. Its core argument: almost anyone can ship an LLM feature, but few can prove it actually works. She identifies four sources of misleading evaluation results and pairs each with a cheap detection habit, complete with references and runnable code (with CLI argument support).

Confirmed

Temperature 0 is not deterministic: Running the same question 1,000 times at temperature 0 can produce 80 different answers (demonstrated with Qwen3), meaning a single evaluation run is unreliable.
Evaluation variance dwarfs typical "gains": The author shows a case where 5 evaluation runs fluctuated between 73% and 93%, warning against mistaking random noise for real improvement.
Benchmark saturation and contamination are widespread: Cited research examined 60 benchmarks and found nearly half already saturated; OpenAI dropped some benchmarks after its frontier models could reproduce (leaked) content from SWE-bench-style benchmarks.
"It works" does not mean task success: One of four common pitfalls in agent evaluation the author highlights. Every tool call can be correct while the task still fails; a correct answer may also come from burning tokens in repetitive loops. The right approach is scoring the full trajectory, covering tool-call correctness, task completion, step efficiency, and plan adherence.
Language edge cases trip up graders: The author cites a Turkish example—Python's "İSTANBUL".lower() does not return "istanbul"—so exact-match graders may wrongly mark correct Turkish answers as incorrect.

Why it matters

Evaluation is the only basis for enterprises to judge whether an LLM feature works, yet temperature-0 nondeterminism and trajectory-blind scoring lead teams to overestimate system reliability and misread iteration effects.
The author's remedies emphasize "cheap": no massive re-runs needed—low-cost habits like multiple sampling, trajectory-level scoring, and language-aware matchers can substantially improve evaluation credibility.

2026-10-30 ~ 2026-10-30 · 5 related posts

Primary sources

"Your LLM evaluation is lying to you": temp-0 Qwen3-235B gives 80 different answers in 1000 runs — irombie · 2026-10-30 19:37 (UTC+08:00)
Your LLM eval is probably lying: 4 failure modes and cheap fixes from an Istanbul talk — irombie · 2026-10-30 18:34 (UTC+08:00)
Your LLM Eval Is Lying: Formatting Swings LLaMA-2 Accuracy by 76 Points — irombie · 2026-10-30 18:39 (UTC+08:00)

[source] Your LLM eval is probably lying: 4 failure modes and cheap fixes from an Istanbul talk — irombie · 2026-10-30
Same Eval, Same Model, 5 Runs: 73% to 93% — Your LLM Eval Is Mostly Variance — irombie · 2026-10-30
[source] Your LLM Eval Is Lying: Formatting Swings LLaMA-2 Accuracy by 76 Points — irombie · 2026-10-30
[source] "Your LLM evaluation is lying to you": temp-0 Qwen3-235B gives 80 different answers in 1000 runs — irombie · 2026-10-30
Python gotcha breaks LLM evals: "İSTANBUL".lower() doesn't return "istanbul" — irombie · 2026-10-30