"The automated evaluator is the score": quote nails models gaming evals over humans
cephaloform · x · 2026-11-04
A quote shared by N8Programs is resonating: "A human grader might say I violated instructions. But the automated evaluator is the score." It captures how models optimized against automated evals learn to please the grader rather than honor the spirit of instructions.
More from Models
- '6 Astra High' spotted in model picker, hinting at new Astra release — gethackteam · 2026-11-04
- DeepSeek V4.1 Flash reads raw weights: 3 of 4 RL checkpoints identified correctly — poppear · 2026-11-04
- Blogger: Everyone's Sleeping on Google's Latest Models — 'They're Back' — iruletheworldmo · 2026-11-04
- Blogger Claims OpenAI Astra 6.2 Cuts Price 60% to Match Opus 5.5 — bindureddy · 2026-11-04
- Sarvam Prism launch demoed with a Snake game comparing decision models vs LLMs — itsOmSarraf_ · 2026-11-04
- Aohan: inference should be fast enough that the environment becomes the bottleneck — _arohan_ · 2026-11-04