Hidden-bug eval across 105 issues: Fable 5.1 finds 43, none fixes all — cost per model compared
PawelHuryn · x · 2026-09-03
PawelHuryn ran an agent bug-hunting eval on the Antigravity CLI using 2 real repos with 105 hidden bugs:
- Fable 5.1 (max): 43/105 for $77.55
- GPT-5.6 (max): 42/105 for $69.61
- Grok 4.6 (xhigh): 27/105 for $16.96
- Opus 5 (max): 21/105 for $51.33
- Gemini 3.8 Flash (high): 20/105 for $9.78
Gemini 3.8 Flash is a clear jump from 3.6 Flash — not the top, but strong value. These aren't random bugs but hard issues agents still miss in early 2026; "unplanted" issues are excluded since models commonly report and fix real but irrelevant issues.
43 bugs remain unfixed by any model. Practical tip: verify your work with a different model family — same-family models share blind spots.
Related event: Bug Hunt Bench: GPT-6 Astra tops 105 real-bug test at 48/105(11 posts)→
More from coding & agent
- Burkov: if you don't interrogate Codex line-by-line at handoff, you get 80%-done work — burkov · 2026-10-21
- Mintlify launches Autopilot, an always-on agent that updates docs as you ship — stuffyokodraws · 2026-10-21
- Skills vs MCP vs RAG vs Memory: the four knowledge layers production agents need — MaryamMiradi · 2026-10-21
- When customer service bans AI agents, one goes incognito to fight Xfinity — MoonL88537 · 2026-10-21
- Every.to publishes raw interviews so readers can agent-check quotes in its Codex story — every · 2026-10-21
- One-prompt Blender: Voyager + Claude Opus 5.5 auto-connects to Blender and Unity — ravisparikh · 2026-10-21