Benchmark hacking? Labs' own SWE-bench scores beat independent runs by 4-5 points on average

RexDouglass · x · 2026-11-18

SolomonMg defines 'benchmark hacking' (analogous to p-hacking): testing many model versions with slightly different setups and publishing only the best score, portraying noise as signal. The test: on prized benchmarks, labs' self-reported scores should exceed independent runs — and on SWE-bench Verified they do, beating Epoch AI's and the SWE-bench team's reproductions by 4-5 points on average. A chart breaks this down per model: Claude Opus 4.7 (Epoch +4.1), GPT-5.2 (Epoch +6.2, mini +7.2), Kimi K2.6 (Epoch +3.5), Gemini 3 Pro (Epoch +3.3, mini +6.6), GLM-5 (Epoch +5.7), with outliers like Qwen3.6-Plus at Epoch +20.9. The author notes unique harnesses and settings likely explain much of the gap — not necessarily deliberate gaming.

Original post →

More from Models

Models channel →