Sonnet 5.5 shows rare judge disagreement, Cohen's κ of just 0.16 vs GPT judges on reward hacking
dhadfieldmenell · x · 2026-10-14
A new evaluation of LLM-as-judge reliability found Claude Sonnet 5.5 agreeing with other model judges far less than usual. On 200 trajectories measured with Cohen's κ, Sonnet 5.5 scored just 0.16 against both GPT-6 Luna and GPT-6 Astra, and 0.33 vs Opus 5.5 — while other judge pairs ranged from 0.4 to 0.8.
The specific issue: Sonnet 5.5 is dramatically more lenient on what counts as reward hacking. This is a direct warning for anyone using Sonnet 5.5 as an eval judge — it may systematically under-flag reward hacking in agent evaluations.
More from Models
- Argon model 'incredibly cooked': skips reasoning and leaks its chain-of-thought as plain text — scaling01 · 2026-10-14
- Fields Medalist Gowers says Grok solved a graph theory problem he studied for a year — paul_cal · 2026-10-14
- LiquidAI's edge-focused MoE LFM2.5-24B-A2B trends on Hugging Face — LiquidAI · 2026-10-14
- Gemini 4 Argon goes GA in GitHub Copilot, beating prior Gemini models on coding — 0xkarasy · 2026-10-14
- Gemini 4 Argon lands in Cursor: 52.8% on CursorBench, matching Opus 5.5 at $2/$10 — gaganghotra_ · 2026-10-14
- User reports Claude is far less deceptive, nearing Codex-level attention to detail — DevDminGod · 2026-10-14