Repeating LLM Evals Doesn't Boost Accuracy Due to Correlated Errors
randal_olson · x · 2026-07-30
Many developers run LLM evals multiple times and take the majority vote to increase reliability. However, a new analysis by Ege Altan reveals that this practice offers limited benefits.
The core issue is error correlation: LLM judges tend to fail on the exact same traces repeatedly. If a judge has 80% accuracy and its errors were purely random, running it 9 times should theoretically push accuracy near 98%. But because errors are correlated, the actual accuracy stalls around 84.5%.
The article uses simulations to demonstrate how correlated errors undermine the value of repeated evals, noting that consensus from different models provides little new information. To truly trust your evals, incorporating ground truth human labels remains the most effective solution.
Related event: Study Reveals Majority Voting is Ineffective for LLM Evaluation(2 posts)→
More from coding & agent
- 'LLM code is sloppy' rebutted with real ffmpeg code: identical if/else branches in a top library — kingslyj · 2026-10-16
- Agent actions need checks outside the model: NVIDIA demos lean on guardrails amid wave of real agent incidents — Creamy-And-Crowded · 2026-10-16
- Vitalik builds "paranoid mode" agent wrapper with Tor, zkAPI and local prompt rewriting — DavideCrapis · 2026-10-16
- Agent fine-tunes Qwen3.5-0.8B with synthetic data: 4.7%→76.3% accuracy, 11.9× faster — ben_burtenshaw · 2026-10-16
- Beam-501B agent demo: solo beats 5-agent team on ProgramBench rebuild task — ben_burtenshaw · 2026-10-16
- Beam agent builds BeamSpeak, a local Parakeet dictation app for macOS — ben_burtenshaw · 2026-10-16