Bug Hunt Bench: GPT-6 Astra fixes 45 of 105 planted bugs; Opus 5.5 ties at 43
PawelHuryn · x · 2026-09-23
Pawel Huryn planted 105 real bugs across two repos and blind-graded frontier coding models via judges from different model families against a secret answer key. Results: GPT-6 Astra (max) fixed 45 for $33.03 (n=3); Fable 5.1 (max) 43 for $77.55; Opus 5.5 (max) 43 for $60.49; Opus 5.0 (max) only 27; Muse Spark 1.3 (max) 32.2 for $18.11. Unplanted or irrelevant OpenAI-reported bugs don't count. Cost spread across the board is roughly 200x. Takeaway: Anthropic is back in the game, with huge per-model price/performance differences. Full leaderboard is live on Bug Hunt Bench.
Related event: Bug Hunt Bench: GPT-6 Astra Tops 105-Bug Fixing Test, MiMo Stuns on Value(18 posts)→
More from coding & agent
- As models improve, writing good evals gets harder and more important — amankhan · 2026-11-12
- Omarchy 4.6 lets you share local AI models across your tailnet, says DHH — alexcovo_eth · 2026-11-12
- Coding harnesses hand shell access to inference providers — 4-step gateway mitigation — shensi · 2026-11-12
- Slack's David Hoang: agents beat people on coordination, proposes 'boundary agents' — davidhoang · 2026-11-12
- CS student mourns coding flow states he'll never experience in the AI era — chrissyykat · 2026-11-12
- AI Engineer Code Summit features its first all-female keynote lineup — altryne · 2026-11-12