Same Eval, Same Model, 5 Runs: 73% to 93% — Your LLM Eval Is Mostly Variance
irombie · x · 2026-10-30
The author breaks down a common eval pitfall: "It works" doesn't mean the task succeeded. All tool calls can be correct while the task still fails, and right answers can come from loops and burned tokens. The fix: score the whole trajectory — tool correctness, task completion, step efficiency, and plan adherence.
The bigger issue is variance: same eval, same model, 5 runs, results from 73% to 93%. Temperature 0 doesn't save you either — 1,000 samples at temp 0 produced 80 different answers. The fix: rerun and report the spread; a change smaller than the spread is variance, not a result.
Related event: Your LLM Evaluation May Be Lying: High Variance and Contaminated Benchmarks(3 posts)→
More from coding & agent
- Can You Pass This 5-Question PyTorch Interview Speedrun? eval, zero_grad, detach and More — cneuralnetwork · 2026-10-30
- Full AI Short Film Workflow: Character/Props/Location Sheets + Seedance Reference Mode — socialwithaayan · 2026-10-30
- David Khourshid's top prompting trick: append 'Ask Fable what it thinks' — DavidKPiano · 2026-10-30
- Harness launches E2E-encrypted orchestration for AI coding agents across your machines — dee_hw · 2026-10-30
- 3 PCs, 120+ threads, 36 coding agents: he says local setups need CPUs, not GPUs — kevinnbass · 2026-10-30
- Company rushes AI agent across all internal systems, now worries it sees too much private data — Different_Pain5781 · 2026-10-30