Bug Hunt Bench: GPT-6 Astra has unique blind spots, prompting a new model-routing strategy
PawelHuryn · x · 2026-09-05
PawelHuryn's Bug Hunt Bench — blind-graded runs of frontier coding models against 105 real planted bugs in repos — shares new data and FAQ:
- Key finding: GPT-6 Astra has completely different blind spots from other models, leading to a new routing strategy — implementation with Astra, review with Fable or Sol.
- Methodology notes:
- Unplanted bugs don't count; OpenAI models report many real-but-irrelevant issues ("bug-maxing")
- All bugs were missed by frontier models in early 2026
- Answer keys stay private, which is why the benchmark works
- Costs span 200x (log axis); run times span under 7x
- The harness will be open-sourced so others can test with their own repos; data and JSON are public.
Related event: Bug Hunt Bench: GPT-6 Astra tops 105 real-bug test at 48/105(11 posts)→
More from coding & agent
- Skills vs MCP vs RAG vs Memory: the four knowledge layers production agents need — MaryamMiradi · 2026-10-21
- When customer service bans AI agents, one goes incognito to fight Xfinity — MoonL88537 · 2026-10-21
- Every.to publishes raw interviews so readers can agent-check quotes in its Codex story — every · 2026-10-21
- One-prompt Blender: Voyager + Claude Opus 5.5 auto-connects to Blender and Unity — ravisparikh · 2026-10-21
- Aviation tracks every part's provenance — agents can reason over that graph — colinmcnamara · 2026-10-21
- Claude Code directed a full music video: 6.5 hours, 765M input tokens — colinmcnamara · 2026-10-21