105 real bugs benchmarked: Sonnet 5.5 max scores 55.5, beating GPT-6 Astra at 45
PawelHuryn · x · 2026-09-29
PawelHuryn ran a bug-hunting benchmark on 2 real repos with 105 planted bugs. Results: Sonnet 5.5 (max) 55.5, GPT-6 Astra (max) 45, GPT-5.6 Sol (max) 43.5, Fable 5.1 (max) 43, Opus 5.5 (max) 41.7, Sonnet 5.5 (xhigh) 39. The key finding: Sonnet 5.5 max wins by being the least lazy model — it used 1,330 turns (6× Astra's 222, 3× Opus 5.5's) to fix 10–14 more bugs. The xhigh mode halved turns but dropped below Opus 5.5. Runs were validated with mutation testing (400 mutants, 2h23m per repo); the author plans at least 2 repeats per config.
More from coding & agent
- Google Research open-sources PaperOrchestra, a multi-agent framework for writing AI papers — google-research · 2026-11-16
- ffmpeg-skill gives coding agents local video editing via 42 FFmpeg tools and MCP — kajisho5 · 2026-11-16
- Agents Now Reply to Slack Messages on Their Own, Resolving a Prod Incident Ghost-to-Ghost — iamrobotbear · 2026-11-16
- Engineer's Interview Prep Method: Use Claude to Find Which Answers Undersell Your AI Work — brandon_galang · 2026-11-16
- DeepSeek open-sources its app frontend "chat-demo", already at 3.9k stars — NielsRogge · 2026-11-16
- lemire: AI's real impact on coding is expanding the engineer's 'cognitive radius', not speed — lemire · 2026-11-16