Bug Hunt Bench ranks GPT-6 Astra top as coding models fix real planted bugs, costs spread 200x
PawelHuryn · x · 2026-09-23
Pawel Huryn launched Bug Hunt Bench, a blind-graded leaderboard pitting frontier coding models against 105 real planted bugs (one prompt per repo, judges from different model families scoring against a secret answer key).
- Current ranking: GPT-6 Astra > GPT-5.6 Sol ≈ Opus 5.5 > Fable 5.1 > Opus 5
- Only pre-planted bugs count; unplanted bug reports (notably from OpenAI's "bugmaxing") are excluded, which the author argues is arguably negative
- Costs across runs span roughly 200x (log axis); time spread is under 7x
- Next up: n=3 run for Opus 5.5 and a separate GPT-6 Sol post
Related event: Bug Hunt Bench: GPT-6 Astra Tops 105-Bug Fixing Test, MiMo Stuns on Value(18 posts)→
More from coding & agent
- YC-backed Roma turns your to-do list into an agent command center across Gmail and Notion — testingcatalog · 2026-11-12
- Boubou: open-source CLI turns your laptop into a fully local voice AI hub — alexcovo_eth · 2026-11-12
- User automates entire video clipping workflow with Nous Research's Hermes agent — Teknium · 2026-11-12
- Tool-call failure rates jump 14x across providers, charts pinpoint inference stack updates — remilouf · 2026-11-12
- As models improve, writing good evals gets harder and more important — amankhan · 2026-11-12
- Prime agent ships with day-one multi-agent communication, runs in Minecraft unmodified — sarahcat21 · 2026-11-12