Benchmaxxing is mostly training focus, not cheating, argues nrehiew_ on FrontierCode
nrehiew_ · x · 2026-10-29
Researcher nrehiew pushes back on claims that FrontierCode 2.0's results (only Claude Opus 5.5 at 45.5% and GPT-6 Astra at 42.8% scoring well) prove benchmaxxing:
- Modern benchmarks measure narrow capabilities: DeepSWE (implement a given spec), FrontierCode (devise the spec yourself and implement), FrontierSWE (long-horizon tasks). Nobody picks models off leaderboards anymore; preference is subjective.
- A popular benchmark signals a domain people care about; labs post-train for those domains. A gap on a newer benchmark likely means the lagging lab didn't post-train that capability.
- Training on test is now too costly reputationally for serious players—it was a paper-publishing-era phenomenon. Opus 5's strong implementation but weak communication shows differing post-training priorities, not cheating.
- He still favors private test sets.
More from Models
- Brin's hands-on push and Google's code red may be paying off — Argon next — haider1 · 2026-10-29
- ChatGPT users report Computer history broken for weeks despite being on — RileyRalmuto · 2026-10-29
- You.com: agentic search failures are behavior problems, fixed in Nemotron 3.5 Lightning — RichardSocher · 2026-10-29
- How I AI agent bench: Grok Bot wins big at 93%, Claude Duet trails at 79% — lennysan · 2026-10-29
- DeepMind slammed for announcing models weeks before availability amid Gemini Xenon chatter — adonis_singh · 2026-10-29
- Alphabet Q3 call: Gemini Carbon frontier model, MCP apps in Search, AI Mode ads expand — lilyraynyc · 2026-10-29