Study finds LLM judges worthless below moderate agreement
IanArawjo · x · 2026-08-19
A study using extensive simulations determined the effectiveness thresholds for LLM-as-a-judge. It found that below 'moderate agreement,' LLM judges are not worth the trouble. However, achieving 'substantial' agreement yields gains of 1.5x and above.
More from Research
- Ex-OpenAI researcher claims most neurons are legible now; critics call it hype — anshulkundaje · 2026-10-16
- Physicist pushes back on ML community's acceptance that models may be beyond understanding — tszzl · 2026-10-16
- Survey: AI agent evaluation is getting realistic but still fails to measure safety and reliability — burkov · 2026-10-16
- COVA-SFT dataset trains models to solve text problems with visual chain-of-thought — yoavartzi · 2026-10-16
- USTC paper uses speculative sampling to cut latency in interactive video generation — ustc · 2026-10-16
- Tsinghua's CoTranslator distills reverse CoT to boost 4B model math scores — Tsinghua · 2026-10-16