Study Finds Majority Voting for LLM Evals Ineffective; Human Labels Still Essential
randal_olson · x · 2026-07-30
A study on LLM evaluation methodologies reveals that the common trick of running an eval multiple times and taking the majority answer yields disappointing results.
Theoretically, if a grader has an 80% accuracy, running it 9 times with majority voting should boost accuracy to 98%. However, experiments show that actual accuracy only reaches 84.5% because the grader consistently makes mistakes on the exact same edge cases. The author emphasizes that human labels remain the definitive solution and provides a method to calculate the required volume of human annotation.
Related event: Study Reveals Majority Voting is Ineffective for LLM Evaluation(2 posts)→
More from Research
- Goodfire's Nick Cammarata argues InceptionV1 polysemanticity wasn't caused by superposition — petrusenko_max · 2026-10-16
- LLM ideation experiment: just ask for non-obvious ideas — and Opus beats smaller models by a wide margin — arjunrajlab · 2026-10-16
- SimpleMark: multi-bit LLM watermark bounds per-token distortion with f-divergence budget — rishabh16_ · 2026-10-16
- Stanford paper: generative representation learning cuts tokenizer training compute by 7-8x — HirokatuKataoka · 2026-10-16
- Cohere Labs to present confidence estimation for reasoning models at EMNLP 2026 — Cohere_Labs · 2026-10-16
- Vals RSI Index extrapolates Google models hitting 130% autonomous AI R&D by Oct 2027 — HaydnBelfield · 2026-10-16