MaxxBench: A Simple Filename Case Change Exposes DeepSWE's Top Models
xeophon · x · 2026-11-08
The new benchmark MaxxBench takes DeepSWE's 89 coding tasks and makes only trivial changes to the instructions or starter code — such as lowercasing a filename. Models that originally clustered around 74% now spread across a gap of up to 23 percentage points, suggesting many models are merely "good at gaming DeepSWE" rather than genuinely good at coding. The benchmark is designed to test robustness to mundane engineering constraints; a reposter dubbed it a "midwit detector."
More from Models
- Leaked Qwen4-27B early build: ~35GB, tops benchmark, may be sparse MoE — teortaxesTex · 2026-11-08
- Muse token pricing: 2B tokens for $35, plus 1B-token referral code — RichardsonDx · 2026-11-08
- Same model, same config: Reddit user documents rare episodes where agents act like a lousy support rep — No_Dragonfruit_1599 · 2026-11-08
- Unverified report: DeepSeek V4 Pro quietly updated, hard agent task pass rate jumps 16 points — teortaxesTex · 2026-11-08
- User frustrated as Anthropic monthly promo credits expire within 30 days — arthurcolle · 2026-11-08
- Dev calls out Anthropic hype: Boris Cherny claimed coding was 'solved' in February amid buggy Claude Code — kuza55 · 2026-11-08