MaxxBench: models clustered at 74% on DeepSWE spread 23 points when tasks change slightly
dyn___ · x · 2026-11-08
MaxxBench reruns 89 DeepSWE coding tasks with small changes to instructions or starting code — same engineering problems, slightly different requirements. The result: models clustered around 74% on the original leaderboard end up 23 percentage points apart. On the chart, Claude Opus 5.5 tops MaxxBench at 76.4% with little degradation, followed by GPT-6 Astra at 71.9%, while GLM-5.3 (56.7%), DeepSeek V4.1 Flash (55.6%), Muse Spark 1.2 (38.8%) and DeepSeek V4 Flash (28.1%) lose far more — quantifying how much leaderboard performance is benchmaxing.
More from Models
- Flash-sized models are less robust across vendors, per srchvrs analysis — srchvrs · 2026-11-08
- New benchmark: big established models are robust, newcomers and flash versions benchmaxx — srchvrs · 2026-11-08
- Flash-sized models are less robust across the board, benchmark thread finds — srchvrs · 2026-11-08
- AI auditors catch 98% of agent misbehavior but drown reviewers in false alarms, study finds — The Decoder · 2026-11-08
- Rewritten DeepSWE trips GLM and DeepSeek, pointing to evaluation-aware benchmarks — teortaxesTex · 2026-11-08
- AI models fade and get lazy over time — dev calls for a Proof of Model standard — sull · 2026-11-08