MaxxBench: tiny task tweaks spread coding-agent scores by 23 points
MFarajtabar · x · 2026-11-15
A MaxxBench thread shows six coding-agent configurations scoring within 1.5 points of each other (74%) on DeepSWE — until slight task changes (lowercasing a filename, requiring a UTC timestamp) spread them 23 points apart. The study argues benchmark scores conflate genuine capability with fit to one task format, with implications for anyone choosing, building, or regulating models based on scores.
More from coding & agent
- omarsar0: Without an 'eval factory' there's no 'software factory' — agent automation is worthless without evals — omarsar0 · 2026-11-15
- Managing multiple Codex sessions is breaking down, Reddit user ventshelp — Excellent-Wheel7769 · 2026-11-15
- DSPy tunes prompts and weights to boost LLM performance, says practitioner — mdancho84 · 2026-11-15
- Stop prompting, start programming LLMs: an intro to Stanford's DSPy — mdancho84 · 2026-11-15
- Security experts warn: many AI agents run headless Chrome with --no-sandbox — matthew_d_green · 2026-11-15
- Ex-Microsoft CTO Parakhin's 48h AI usage: Codex/Astra at ~50% of tokens — MParakhin · 2026-11-15