Bug Hunt Bench: Sonnet 5.5 Leads at 55.5 Points but at High Cost and Variance

Pawel Huryn (author of The Product Compass) launched and continues to update Bug Hunt Bench: 105 real bugs from open-source repositories that frontier models failed to find in early 2026 are replanted, and models are evaluated blind with one prompt per repo on finding and fixing them. In the first comparison, Sonnet 5.5 (max) scored 55.5, far ahead of GPT-6 Astra and other flagships — but high scores come with high cost and variance.

Confirmed

Why it matters

2026-09-29 ~ 2026-09-30 · 5 related posts

Full story(2 episodes)→

Primary sources