Benchmark hacking? Labs' own SWE-bench scores beat independent runs by 4-5 points on average
RexDouglass · x · 2026-11-18
SolomonMg defines 'benchmark hacking' (analogous to p-hacking): testing many model versions with slightly different setups and publishing only the best score, portraying noise as signal. The test: on prized benchmarks, labs' self-reported scores should exceed independent runs — and on SWE-bench Verified they do, beating Epoch AI's and the SWE-bench team's reproductions by 4-5 points on average. A chart breaks this down per model: Claude Opus 4.7 (Epoch +4.1), GPT-5.2 (Epoch +6.2, mini +7.2), Kimi K2.6 (Epoch +3.5), Gemini 3 Pro (Epoch +3.3, mini +6.6), GLM-5 (Epoch +5.7), with outliers like Qwen3.6-Plus at Epoch +20.9. The author notes unique harnesses and settings likely explain much of the gap — not necessarily deliberate gaming.
More from Models
- Paying users complain: model tuned for 980M free users leaves subscribers with 'scraps' — Yamapama · 2026-11-18
- Users rip ChatGPT voice mode: random interruptions, accent drift it can't fix — Yamapama · 2026-11-18
- Oru'el claims Kanook beats NVIDIA Flora and OpenAI on RoboRewardBench by 30-50% — AashaySachdeva · 2026-11-18
- Flash model size chart puts Qwen3.8-Flash-Next smallest at 176B total parameters — BerryBlush_Rogers691 · 2026-11-18
- Agent-tuned MiMo scores 0/10 in a real loop while base Qwen 3.5 gets 7-8/10 — GapNew4766 · 2026-11-18
- Tencent releases Hy4 flash: matches Claude Haiku 5.5 (high) at 18x lower task cost — ahuja_priyank · 2026-11-18