105 planted bugs: Fable 5.1 fixes 43 for $77.55, Grok 4.6 gets 27 for just $16.96
pakupdates85 · x · 2026-09-03
Testing frontier coding models on Bug Hunt Bench (2 real repos, 105 hidden bugs, one prompt per repo):
- Fable 5.1 (max): 43/105, $77.55
- GPT-5.6 (max): 42/105, $69.61
- Grok 4.6 (xhigh): 27/105, $16.96
- Opus 5 (max): 21/105, $51.33
- Gemini 3.8 Flash (high): 20/105, $9.78
Gemini 3.8 Flash isn't top but clearly jumps from 3.6 Flash's 16/20 at the lowest cost. 43 bugs remain unfixed by any model; costs vary nearly 8x across runs with weak correlation to bugs fixed.
Related event: Bug Hunt Bench: GPT-6 Astra tops 105 real-bug test at 48/105(11 posts)→
More from coding & agent
- Skills vs MCP vs RAG vs Memory: the four knowledge layers production agents need — MaryamMiradi · 2026-10-21
- When customer service bans AI agents, one goes incognito to fight Xfinity — MoonL88537 · 2026-10-21
- Every.to publishes raw interviews so readers can agent-check quotes in its Codex story — every · 2026-10-21
- One-prompt Blender: Voyager + Claude Opus 5.5 auto-connects to Blender and Unity — ravisparikh · 2026-10-21
- Aviation tracks every part's provenance — agents can reason over that graph — colinmcnamara · 2026-10-21
- Claude Code directed a full music video: 6.5 hours, 765M input tokens — colinmcnamara · 2026-10-21