FULL STORY

Bug Hunt Bench: Frontier Models Put to the Bug-Fixing Test

Pawel Huryn launched Bug Hunt Bench, testing frontier models on real open-source bug fixes. Anthropic's new models set records, with Opus 5.5 leading though at high cost.

2026-09-29 ~ 2026-09-30 · 2 episodes · 8 posts

Episode 1 · Anthropic's New Models Set Records on Bug Hunt Bench (2026-09-29, 3 posts)

Pawel Huryn reports that Anthropic's new models set new records on Bug Hunt Bench, with Opus 5.5 (max) making 138–163 calls and Sonnet 5.5 performing 628 calls over 111 minutes of self-checking. He later corrected the data, noting the gap between the two models is even larger than first described.

Episode 2 · Bug Hunt Bench: Sonnet 5.5 Leads at 55.5 Points but at High Cost and Variance (2026-09-29, 5 posts)

Pawel Huryn (author of The Product Compass) launched and continues to update Bug Hunt Bench: 105 real bugs from open-source repositories that frontier models failed to find in early 2026 are replanted, and models are evaluated blind with one prompt per repo on finding and fixing them. In the first comparison, Sonnet 5.5 (max) scored 55.5, far ahead of GPT-6 Astra and other flagships — but high scores come with high cost and variance.

Confirmed

  • First comparison (scored by number of bugs fixed): Sonnet 5.5 (max) 55.5, far ahead of GPT-6 Astra and other flagship models
  • Sonnet 5.5 effort-level tests (average n=2): max 55.5 points, 1330 turns, 234.7 minutes, $134.79; xhigh 39 points
  • The benchmark added a log-scale "score vs. turns" view with Sonnet 5.5 data across effort levels
  • Three long autonomous-agent runs of Sonnet 5.5 (max) on two repos varied notably: Run 1 scored 57 (1,566 turns, 258 minutes, $154.05); Run 2 scored 54; Run 3 hit 1,833 turns, up to 6.5 hours, and $192

Why it matters

  • The benchmark uses real repo bugs rather than synthetic tasks, closer to actual engineering debugging
  • High scores come with high cost and variance: single runs often cost $100+ and hours, so model selection should weigh cost-efficiency and stability, not just scores