LLM judges err in correlated ways, breaking standard benchmark aggregation

burkov · x · 2026-10-19

The article tackles a practical flaw in AI evaluation: when multiple LLM judges score the same item, standard aggregation assumes independent errors, yet LLM judges share training data, architectures, prompt templates, and failure modes, so their votes are correlated.
Ignoring this correlation makes evaluations overconfident and even confidently wrong — risky as benchmarks increasingly steer model selection, safety decisions, and policy.
The author proposes and tests dependence-aware aggregation that explicitly models pairwise agreement/disagreement among judges, comparing it against conditional-independence baselines like uniform majority vote for inferring the true label.

More from Research