Benchmark audit: 5 of 6 AI benchmarks have grading defects; only PostTrainBench verified
maksym_andr · x · 2026-11-10
Open Benchmark Review audited several AI benchmarks: only PostTrainBench passed, with no score-affecting grading errors across its 24 tasks and no reward hacks in top unflagged runs. Confirmed grading-defect rates: HackBench 63.3%, AI4AI-Bench 40%, DeepRSI and FrontierML 33.3% each, FrontierCS 2 31.2% (vs. a 20% flaw threshold). Many popular benchmark scores may be unreliable.
More from Models
- System card confirms Claude MyTHOS 5 and Fable 5 share the same underlying weights — repligate · 2026-11-10
- Early Tester: Fable 5.2 Built a 3,100-City Water Map in One Go, but Still Makes Silly Errors — alliekmiller · 2026-11-10
- Fable 5.2 burns so many tokens its 20% price cut is effectively a wash — addyosmani · 2026-11-10
- Opus formalizes CPython in 613K lines of Lean — and finds real Python bugs — pbaylies · 2026-11-10
- CursorBench 4.0: Fable 5.2 High effort hits 56.9% at $9.70 per task — SEO · 2026-11-10
- Rule of thumb: use Opus 5.5 for tasks you know, Fable 5.2 for creative exploration — EricBuess · 2026-11-10