Study: LLM Judges Are Useless Below Moderate Agreement
IanArawjo · x · 2026-08-19
- Core Question: How reliable must an LLM judge be to be useful?
- Findings: Extensive simulations reveal that LLM judges are not worth the trouble if they fall below "moderate agreement." Achieving "substantial agreement" yields gains of 1.5x or more.
- Metrics: It is recommended to report IRR metrics to ensure evaluation quality.
More from Research
- LMR-BENCH at EMNLP 2025 tests whether LLM agents can reproduce real LLM research — ZhiyuChen4 · 2026-10-16
- Ex-OpenAI researcher claims most neurons are legible now; critics call it hype — anshulkundaje · 2026-10-16
- Physicist pushes back on ML community's acceptance that models may be beyond understanding — tszzl · 2026-10-16
- Survey: AI agent evaluation is getting realistic but still fails to measure safety and reliability — burkov · 2026-10-16
- COVA-SFT dataset trains models to solve text problems with visual chain-of-thought — yoavartzi · 2026-10-16
- USTC paper uses speculative sampling to cut latency in interactive video generation — ustc · 2026-10-16