AI Judges Overestimate New Models and Miss Hidden Flaws
rohanpaul_ai · x · 2026-07-06
Analysis indicates that while using AI as a judge (LLM-as-judge) can directionally track model progress, it often misses hidden quality issues that human evaluators would catch.
Automated judges showed only about a 3% agreement rate with humans on early models but severely overestimate newer ones: the human pass rate for GPT-5.5 was inflated from 6.25% to 17.9%, and Opus 4.8 from 8.33% to 18.8%. This suggests that automated evaluations may suffer from systematic distortion when assessing new models.
More from Models
- Fern's "context is intelligence" thesis: prediction keeps improving out to 10.2M tokens — bneyshabur · 2026-10-26
- Mirendil launches Fern, a from-scratch pretrained model with up to 10M-token context — bneyshabur · 2026-10-26
- Users spot a dedicated translation UI appearing in Claude — BLCNYY · 2026-10-26
- GrokBot user: flight booking and shopping all blocked, agent can't do anything useful — lucasmeijer · 2026-10-26
- Vending-bench-style eval: Sol 5.6 appears the most aggressive contract-violator — xuanalogue · 2026-10-26
- Fable 5.5 launch rumors persist for third week, but no frontier model drops this month — patience_cave · 2026-10-26