MaxxBench criticized as cargo-cult science: overfitting confounds claimed benchmaxxing
teortaxesTex · x · 2026-11-09
The author doubles down on criticizing the MaxxBench study's design and narrative:
- Opus 5.5 is genuinely much stronger at coding than similarly scoring models, but the study doesn't show it
- It's possible every recent model overfit on the bench with an innate ceiling of 75, and only strong models avoid being distracted by priors — yet the study never demonstrates benchmaxxing in the sense of higher pass rates than on equivalent-difficulty non-memorized problems
- A proper control could be built by testing top models preceding DeepSWE, e.g., GPT 5.5 and Opus 4.7
- Verdict: bad research design and bad narrative — "cargo cult science"; people should read articles, not just authors' framing and figures
The bulk of the effect is plain overfitting (the "the doctor is the mother" kind), not capability collapse or benchmaxxing.
More from Models
- GLM matches Opus at mass-converting Windows drivers into LPE 0days, says security researcher — MalwareTechBlog · 2026-11-09
- Thread: OpenAI's own report shows models' 80% time horizon on AI R&D is just 15 minutes — tobyordoxford · 2026-11-09
- Opus 5.5 bypassed a YubiKey by pushing over HTTPS with a saved OAuth token — appenz · 2026-11-09
- Rumor: Nvidia to pour massive compute into Arete-27B model — generativist · 2026-11-09
- Gemini models keep blowing past length limits while Opus 5.5 over-refuses — jessi_cata · 2026-11-09
- Continual Learning Touted as December's Biggest AI Story as Third Moment Runs 10,000 Agents — haider1 · 2026-11-09