Bug Hunt Bench: Sonnet 5.5 Leads at 55.5 Points but at High Cost and Variance
Pawel Huryn (author of The Product Compass) launched and continues to update Bug Hunt Bench: 105 real bugs from open-source repositories that frontier models failed to find in early 2026 are replanted, and models are evaluated blind with one prompt per repo on finding and fixing them. In the first comparison, Sonnet 5.5 (max) scored 55.5, far ahead of GPT-6 Astra and other flagships — but high scores come with high cost and variance.
Confirmed
- First comparison (scored by number of bugs fixed): Sonnet 5.5 (max) 55.5, far ahead of GPT-6 Astra and other flagship models
- Sonnet 5.5 effort-level tests (average n=2): max 55.5 points, 1330 turns, 234.7 minutes, $134.79; xhigh 39 points
- The benchmark added a log-scale "score vs. turns" view with Sonnet 5.5 data across effort levels
- Three long autonomous-agent runs of Sonnet 5.5 (max) on two repos varied notably: Run 1 scored 57 (1,566 turns, 258 minutes, $154.05); Run 2 scored 54; Run 3 hit 1,833 turns, up to 6.5 hours, and $192
Why it matters
- The benchmark uses real repo bugs rather than synthetic tasks, closer to actual engineering debugging
- High scores come with high cost and variance: single runs often cost $100+ and hours, so model selection should weigh cost-efficiency and stability, not just scores
2026-09-29 ~ 2026-09-30 · 5 related posts
- Episode 1: Anthropic's New Models Set Records on Bug Hunt Bench(2026-09-29, 3 posts)
- Episode 2: Bug Hunt Bench: Sonnet 5.5 Leads at 55.5 Points but at High Cost and Variance(2026-09-29, 5 posts)
Primary sources
- [source] 105 real bugs benchmarked: Sonnet 5.5 max scores 55.5, beating GPT-6 Astra at 45 — PawelHuryn · 2026-09-29
- [source] Bug Hunt Bench: New Blind-Graded Benchmark Tests Frontier Models on 105 Real Bugs — PawelHuryn · 2026-09-29
- Sonnet 5.5 effort levels tested: max scores 55.5 at $134.79 vs 9 for Sonnet 5 — PawelHuryn · 2026-09-29
- Bug Hunt Bench: Sonnet 5.5 max scores 55.5 but costs $134.79 per run — PawelHuryn · 2026-09-29
- [source] Sonnet 5.5 (max) long-horizon agent runs: 1,833 turns, 6.5 hours, $192 with score dropping — PawelHuryn · 2026-09-30