MaxxBench: Tiny task tweaks leave DeepSWE-tuned coding models 23 points apart
RexDouglass · x · 2026-11-08
MaxxBench reruns 89 DeepSWE coding tasks with small changes to instructions or starting code (e.g. lowercase filenames, UTC timestamps with trailing Z). Models clustered around 74% on DeepSWE end up 23 pp apart, exposing severe benchmark overfitting. Deedy notes the only model that scored higher after tweaks was Opus 5.5.
More from coding & agent
- Dev benchmarks tiny LLMs on agent tasks: 9B passes 2/5, stepping up to 12B — draginol · 2026-11-08
- Engineer proposes CI/CD could become a handoff to a deployment agent — andreisavu · 2026-11-08
- Dev launches Local AI benchmark scoring models by hardware on agent tasks and speed — draginol · 2026-11-08
- Personal AI dot can now read inbox sign-in codes to log in on your behalf — Dimillian · 2026-11-08
- Dev plans eye-tracker to automatically feed agents visual context — johnlindquist · 2026-11-08
- Lattice: open-source 'Cursor for research papers' bundles Claude Code, Overleaf and citations in one app — minchoi · 2026-11-08