MaxxBench: Tiny instruction tweaks split DeepSWE leaders by 23 points
eldonredwards · x · 2026-11-09
Ethan Mollick amplifies MaxxBench, which perturbs 89 DeepSWE tasks with small changes to instructions or starting code (e.g. a lowercase filename). Models clustered around 74% on the original leaderboard end up 23 percentage points apart, questioning whether high DeepSWE scores reflect real engineering ability.
The attached image is Rylan Schaeffer's satirical paper "Pretraining on the Test Set Is All You Need", where a 1M-parameter model pretrained on <100K tokens of benchmark data (phi-CTNL) scores perfectly on every benchmark, mocking benchmark contamination.
More from coding & agent
- Dev renders stunning animation in pure Python with Opus 5.5, zero 3D assets — Eric520CC · 2026-11-09
- REA Author Open-Sources 20+ Repos: An Agent-Powered Reverse Engineering Studio From Apps to Binaries — nikola_mr64990 · 2026-11-09
- Agent autonomously trained and deployed a local labeling model while owner was on vacation — max_paperclips · 2026-11-09
- 'Headless browser' dubbed the creepiest tech term as agents inspect 120 cards — yuwen_lu_ · 2026-11-09
- edge-arm-bridge: mirror Edge profiles to native arm64 Chromium so agents run at native speed — unixterminal · 2026-11-09
- Agent Buddy: an ESP32 animated companion that shows your AI coding agents' live status — DanWahlin · 2026-11-09