MaxxBench slightly tweaks DeepSWE tasks and reshuffles rankings across 12 models
dbreunig · x · 2026-11-08
MaxxBench applied tiny tweaks to DeepSWE tasks (e.g. lowercase filenames, trailing 'z' on timestamps) and re-ran 12 model configurations. Rankings shifted noticeably: Claude Opus 5.5 held first (74.2 → 76.4), but DeepSeek V4 Flash collapsed from 53.3 to 28.1 and GPT-5.6 Luna fell to 46.1.
Key takeaways from dbreunig:
- Be skeptical of popular benchmarks — the only task that matters is your own. Muse Spark is mediocre on both benchmarks yet an excellent driver in his Hermes workflow.
- The deeper risk isn't just Goodhart-style benchmark targeting, but that benchmark patterns get trained into models, biasing them toward benchmark-enforced styles. Popular benchmarks are effectively driving your agent; high-taste benchmarks matter.
- The chart also confirms Claude Opus 5.5 is really good.
More from Models
- 1,400 Claude Code sessions suggest Opus 5.5's reasoning shifted abruptly on Nov 3 — churchkey · 2026-11-08
- Leaked Qwen4-27B early build: ~35GB, tops benchmark, may be sparse MoE — teortaxesTex · 2026-11-08
- Muse token pricing: 2B tokens for $35, plus 1B-token referral code — RichardsonDx · 2026-11-08
- Same model, same config: Reddit user documents rare episodes where agents act like a lousy support rep — No_Dragonfruit_1599 · 2026-11-08
- Unverified report: DeepSeek V4 Pro quietly updated, hard agent task pass rate jumps 16 points — teortaxesTex · 2026-11-08
- User frustrated as Anthropic monthly promo credits expire within 30 days — arthurcolle · 2026-11-08