Bug Hunt Bench: GPT-6 Astra tops 105 real-bug test at 48/105

TL;DR: Pawel Huryn's Bug Hunt Bench (bughunt.productcompass.pm) plants 105 hidden bugs across two real repositories and blind-tests frontier coding models with a single prompt per repo. No model found them all in the first round, where Fable 5.1 (max) led at 43/105 ($77.55); two days later GPT-6 Astra took the top spot at 48/105, reportedly 2x faster than GPT-5.6 Sol and cheaper. Huryn also found Astra's blind spots entirely different from other models and switched to a hybrid routing workflow.

Confirmed

• Test setup: Antigravity CLI, two real repositories, one prompt per repo, blind evaluation
• Fable 5.1 (max): 43/105, $77.55
• GPT-5.6 (max, also called GPT-5.6 Sol): 42/105, $69
• Gemini 3.8 Flash (high): 20/105, $9.78, a clear improvement over earlier versions
• GPT-6 Astra: 48/105, first place; author reports 2x the speed of GPT-5.6 Sol and cheaper than both GPT-5.6 Sol and Fable 5.1
• The leaderboard also covers GLM, DeepSeek and others; per Huryn's reply they just weren't all exported in screenshots
• GPT-6 Astra's blind spots are completely different from other models, so the author now uses hybrid routing (e.g., Astra writes code, Fable reviews)
• Part of the benchmark comes from real vulnerabilities that frontier models initially missed in early 2026; answers are kept private, and unplanted bugs are not counted
• The author observed "bug farming"-style behavior: real early-2026 vulnerabilities were still collectively missed

Unconfirmed

• The author recounted that before Antigravity CLI launched, he ran tests via Gemini CLI specifying 3.7 Flash; the startup info showed 3.7, but every API call was silently routed to gemini-3.5-flash due to a configuration issue (post truncated). This flags a silent-downgrade risk for CLI-based benchmarks; whether earlier results were affected remains unstated

Why it matters

• The benchmark measures real-world debugging on actual repositories, closer to engineering practice than synthetic tasks
• Top scores are close but costs differ sharply: Fable and GPT-5.6 differ by one bug at $69$78, while Gemini 3.8 Flash costs nearly 8x less, useful for budget-sensitive use cases
• The discovery of Astra's distinct blind spots led to the author's hybrid routing practice, suggesting multi-model ensembles can complementarily widen bug coverage
• That early-2026 real vulnerabilities remain collectively missed shows systematic blind spots even among top models
• The Gemini CLI silent-downgrade case is a reminder to verify the actually-called model version before trusting benchmark results
• Third-party accounts like pakupdates85 amplified the results, with cost comparison becoming a focal point of discussion

2026-09-03 ~ 2026-09-05 · 11 related posts

• Gemini 3.8 Flash Fixes 20/105 Real Bugs for $9.78, Big Jump Over 3.6 Flash — PawelHuryn · 2026-09-03
• Hidden-bug eval across 105 issues: Fable 5.1 finds 43, none fixes all — cost per model compared — PawelHuryn · 2026-09-03
• Bug Hunt Bench: Gemini 3.8 Flash jumps to 20/105 real bugs, Fable 5.1 leads at 43 — PawelHuryn · 2026-09-03
• 105 planted bugs: Fable 5.1 fixes 43 for $77.55, Grok 4.6 gets 27 for just $16.96 — pakupdates85 · 2026-09-03
• [source] Bug Hunt Bench blind-tests frontier coding models on 105 planted real-repo bugs — PawelHuryn · 2026-09-03
• Gemini CLI silently swapped 3.7 Flash for 3.5; log your actual API calls — PawelHuryn · 2026-09-04
• GPT-6 Astra fixes 48 of 105 real repo bugs, topping model comparison — PawelHuryn · 2026-09-05
• [source] 105 real bugs benchmarked: GPT Astra fixes 48, beats Fable 5.1 at half the cost — PawelHuryn · 2026-09-05
• GPT-6 Astra has unique blind spots, so this dev routes coding to it and reviews elsewhere — PawelHuryn · 2026-09-05
• [source] Bug Hunt Bench: GPT-6 Astra has unique blind spots, prompting a new model-routing strategy — PawelHuryn · 2026-09-05
• Real-bug benchmark shows frontier models 'bug-maxing': real 2026 flaws still missed — PawelHuryn · 2026-09-05