Bug Hunt Bench: Sonnet 5.5 max scores 55.5 but costs $134.79 per run
PawelHuryn · x · 2026-09-29
Pawel Huryn added a log-scale score-vs-turns view to Bug Hunt Bench, his blind-graded benchmark of frontier coding models on 105 real planted bugs, and published per-effort numbers for Sonnet 5.5:
- max: 55.5, 1,330 turns, 234.7 min, $134.79 (n=2)
- xhigh: 39, 588 turns, $60.30
- high: 32, 294 turns, $16.73
- medium: 18, 112 turns, $8.39
- low: 20, 119 turns, $5.75
For comparison, Sonnet 5 max scored 9 at $17.96. The picture: an extremely motivated model, fast and cheap per turn, but cost scales steeply with effort — max is 3x the score of low at 25x the price. Cost spread across the board is roughly 200x.
More from coding & agent
- Google Research open-sources PaperOrchestra, a multi-agent framework for writing AI papers — google-research · 2026-11-16
- ffmpeg-skill gives coding agents local video editing via 42 FFmpeg tools and MCP — kajisho5 · 2026-11-16
- Agents Now Reply to Slack Messages on Their Own, Resolving a Prod Incident Ghost-to-Ghost — iamrobotbear · 2026-11-16
- Engineer's Interview Prep Method: Use Claude to Find Which Answers Undersell Your AI Work — brandon_galang · 2026-11-16
- DeepSeek open-sources its app frontend "chat-demo", already at 3.9k stars — NielsRogge · 2026-11-16
- lemire: AI's real impact on coding is expanding the engineer's 'cognitive radius', not speed — lemire · 2026-11-16