Repeating LLM Evals Doesn't Boost Accuracy Due to Correlated Errors
randal_olson · x · 2026-07-30
Many developers run LLM evals multiple times and take the majority vote to increase reliability. However, a new analysis by Ege Altan reveals that this practice offers limited benefits.
The core issue is error correlation: LLM judges tend to fail on the exact same traces repeatedly. If a judge has 80% accuracy and its errors were purely random, running it 9 times should theoretically push accuracy near 98%. But because errors are correlated, the actual accuracy stalls around 84.5%.
The article uses simulations to demonstrate how correlated errors undermine the value of repeated evals, noting that consensus from different models provides little new information. To truly trust your evals, incorporating ground truth human labels remains the most effective solution.
Related event: Study Reveals Majority Voting is Ineffective for LLM Evaluation(2 posts)→
More from coding & agent
- Managing Agent State with HTML Boards; Bun Allegedly Rewritten in Rust with Claude — Vjeux · 2026-07-31
- deepagents v0.7 Released: Major Boost in Configurability & Context Efficiency — hwchase17 · 2026-07-31
- Instead of Complaining About Agents, Adapt Your Project to Their Training Env — gandamu_ml · 2026-07-31
- GitHub Discussion: Building Agent Skills & New MCP Release — marlene_zw · 2026-07-31
- Devin Showcases New 'Engineer': 10 PRs Merged in First Week — DevinAI · 2026-07-31
- An Engineering Trick to Adapt ML Agents for the Real World — gandamu_ml · 2026-07-31