Bug Hunt Bench: Gemini 3.8 Flash jumps to 20/105 real bugs, Fable 5.1 leads at 43
PawelHuryn · x · 2026-09-03
Pawel Huryn tested Gemini 3.8 Flash (high) on Bug Hunt Bench (blind-graded, one prompt per repo): 2 real repos, 105 hidden planted bugs.
- Fable 5.1 (max): 43/105 for $77.55
- GPT-5.6 (max): 42/105 for $69.61
- Grok 4.6 (xhigh): 27/105 for $16.96
- Opus 5 (max): 21/105 for $51.33
- Gemini 3.8 Flash (high): 20/105 for $9.78
Key takeaways:
- 3.8 Flash is a clear jump from 3.6 Flash's 16/20 — not top of the board but strong value; costs span roughly 200x across runs
- 43/105 bugs remain unfixed by any model — not random bugs but hard issues agents missed in early 2026; "unplanted" fixes are excluded since OpenAI-family models often report and fix real but irrelevant issues
- Practical tip: review your work with a different model family so blind spots don't overlap
Related event: Bug Hunt Bench: GPT-6 Astra tops 105 real-bug test at 48/105(11 posts)→
More from coding & agent
- Skills vs MCP vs RAG vs Memory: the four knowledge layers production agents need — MaryamMiradi · 2026-10-21
- When customer service bans AI agents, one goes incognito to fight Xfinity — MoonL88537 · 2026-10-21
- Every.to publishes raw interviews so readers can agent-check quotes in its Codex story — every · 2026-10-21
- One-prompt Blender: Voyager + Claude Opus 5.5 auto-connects to Blender and Unity — ravisparikh · 2026-10-21
- Aviation tracks every part's provenance — agents can reason over that graph — colinmcnamara · 2026-10-21
- Claude Code directed a full music video: 6.5 hours, 765M input tokens — colinmcnamara · 2026-10-21