FULL STORY
Bug Hunt Bench: Frontier Models Put to the Bug-Fixing Test
Pawel Huryn launched Bug Hunt Bench, testing frontier models on real open-source bug fixes. Anthropic's new models set records, with Opus 5.5 leading though at high cost.
2026-09-29 ~ 2026-09-30 · 2 episodes · 8 posts
Episode 1 · Anthropic's New Models Set Records on Bug Hunt Bench (2026-09-29, 3 posts)
Pawel Huryn reports that Anthropic's new models set new records on Bug Hunt Bench, with Opus 5.5 (max) making 138–163 calls and Sonnet 5.5 performing 628 calls over 111 minutes of self-checking. He later corrected the data, noting the gap between the two models is even larger than first described.
- Opus 5.5 sets Bug Hunt Bench record with 800+ self-check turns — PawelHuryn · 2026-09-29
- Author corrects Bug Hunt Bench numbers, contrast even starker — PawelHuryn · 2026-09-29
- Sonnet 5.5 Makes 628 Calls, Self-Checks 818 Turns on Bug Hunt Bench — PawelHuryn · 2026-09-29
Episode 2 · Bug Hunt Bench: Sonnet 5.5 Leads at 55.5 Points but at High Cost and Variance (2026-09-29, 5 posts)
Pawel Huryn (author of The Product Compass) launched and continues to update Bug Hunt Bench: 105 real bugs from open-source repositories that frontier models failed to find in early 2026 are replanted, and models are evaluated blind with one prompt per repo on finding and fixing them. In the first comparison, Sonnet 5.5 (max) scored 55.5, far ahead of GPT-6 Astra and other flagships — but high scores come with high cost and variance.
Confirmed
- First comparison (scored by number of bugs fixed): Sonnet 5.5 (max) 55.5, far ahead of GPT-6 Astra and other flagship models
- Sonnet 5.5 effort-level tests (average n=2): max 55.5 points, 1330 turns, 234.7 minutes, $134.79; xhigh 39 points
- The benchmark added a log-scale "score vs. turns" view with Sonnet 5.5 data across effort levels
- Three long autonomous-agent runs of Sonnet 5.5 (max) on two repos varied notably: Run 1 scored 57 (1,566 turns, 258 minutes, $154.05); Run 2 scored 54; Run 3 hit 1,833 turns, up to 6.5 hours, and $192
Why it matters
- The benchmark uses real repo bugs rather than synthetic tasks, closer to actual engineering debugging
- High scores come with high cost and variance: single runs often cost $100+ and hours, so model selection should weigh cost-efficiency and stability, not just scores
- 105 real bugs benchmarked: Sonnet 5.5 max scores 55.5, beating GPT-6 Astra at 45 — PawelHuryn · 2026-09-29
- Bug Hunt Bench: New Blind-Graded Benchmark Tests Frontier Models on 105 Real Bugs — PawelHuryn · 2026-09-29
- Sonnet 5.5 effort levels tested: max scores 55.5 at $134.79 vs 9 for Sonnet 5 — PawelHuryn · 2026-09-29
- Bug Hunt Bench: Sonnet 5.5 max scores 55.5 but costs $134.79 per run — PawelHuryn · 2026-09-29
- Sonnet 5.5 (max) long-horizon agent runs: 1,833 turns, 6.5 hours, $192 with score dropping — PawelHuryn · 2026-09-30