Chart shows LLM judges systematically overrate outputs from stronger models
scaling01 · x · 2026-11-22
A shared chart, "Judges overrate tasks made by stronger models," plots judge-minus-human ratings against the MCI gap between task-making model and judge. When the output comes from a model stronger than the judge, LLM judges inflate scores — e.g. GPT-5 rated a Fable 5.1 task 0.82 vs human 0.26; GPT-6 Astra rated its own output 0.80 vs human 0.30. Ratings track humans closely only when the judge is the stronger model. The poster frames this as relevant to Chinese labs catching up.
More from Models
- Wenhu Chen Revives 'But Can They Reason?' Debate, Asking Why Critics Went Silent — WenhuChen · 2026-11-22
- Gemini 4 Neon hits 59% on rare-disease benchmark, near Claude Fable 5.2 at lower cost — danielmckinn0n · 2026-11-22
- Tesla AI lead: new system uses less compute than the old one did — aelluswamy · 2026-11-22
- JUP-Eval tests whether models hold correct answers under user pushback, not just once — geoffwolfe · 2026-11-22
- Dev praises Gemini's vivid code comments: 'like an open kitchen drawer' — AI_Andrew · 2026-11-22
- Chinese models trail by ~200-250 MCI — and that gap may give them better feedback to catch up — scaling01 · 2026-11-22