Bug Hunt Bench: GPT-6 Astra leads at 45/105, Grok 4.7 scores just 28.7
PawelHuryn · x · 2026-09-22
Pawel Huryn of The Product Compass planted 105 real bugs across two repos and had frontier models find and fix them, launching a blind-graded leaderboard called Bug Hunt Bench. Three-run averages: GPT-6 Astra (max) 45, Muse Spark 1.3 (max) 32.2, Grok 4.7 (xhigh) 28.7 — identical to Grok 4.6, which the author stresses is not a typo — Opus 5 (max) 27, Qwen3.8-Max 25.7. His verdict: "That's not the model we've been waiting for." The site supports score/cost/time comparisons, with cost on a log axis spanning roughly 200x, and full run notes on GitHub.
Related event: Bug Hunt Bench: GPT-6 Astra Tops 105-Bug Fixing Test, MiMo Stuns on Value(18 posts)→
More from coding & agent
- As models improve, writing good evals gets harder and more important — amankhan · 2026-11-12
- Omarchy 4.6 lets you share local AI models across your tailnet, says DHH — alexcovo_eth · 2026-11-12
- Coding harnesses hand shell access to inference providers — 4-step gateway mitigation — shensi · 2026-11-12
- Slack's David Hoang: agents beat people on coordination, proposes 'boundary agents' — davidhoang · 2026-11-12
- CS student mourns coding flow states he'll never experience in the AI era — chrissyykat · 2026-11-12
- AI Engineer Code Summit features its first all-female keynote lineup — altryne · 2026-11-12