Chart shows LLM judges systematically overrate outputs from stronger models

scaling01 · x · 2026-11-22

A shared chart, "Judges overrate tasks made by stronger models," plots judge-minus-human ratings against the MCI gap between task-making model and judge. When the output comes from a model stronger than the judge, LLM judges inflate scores — e.g. GPT-5 rated a Fable 5.1 task 0.82 vs human 0.26; GPT-6 Astra rated its own output 0.80 vs human 0.30. Ratings track humans closely only when the judge is the stronger model. The poster frames this as relevant to Chinese labs catching up.

Original post →

More from Models

Models channel →