Real-world test: GPT-6 Astra tops bug-hunt benchmark with 45.0, finds 105 bugs frontier models missed
PawelHuryn · x · 2026-10-13
PawelHuryn tested frontier models on real work: 2 repos and 105 bugs that frontier models missed in Q1 2026, asking each to find and fix what it could.
Scores (max effort):
- GPT-6 Astra (max): 45.0
- Opus 5.5 (max): 41.7
- Muse Spark 1.3 (max): 32.2
- Muse Wonder 1.0 (max): 31.7
- Grok 4.7 (xhigh): 29.0
The author says he doesn't trust maxed-out benchmarks: across multiple runs, Muse Wonder 1.0 performed similarly to Spark 1.3, was a bit slower, and cost 50% more. All anonymized data is on GitHub; answer keys are not public, and filters can be adjusted to explore more models and effort levels.
More from coding & agent
- Jason Fried: improving existing software with AI has never been more fun — martyamark · 2026-10-14
- TV noise crashes voice agents: task completion falls from 38.7% to 8.6% across 14 stacks — solyarisoftware · 2026-10-14
- Write your own PR descriptions, even when agents write the code — rseroter · 2026-10-14
- AI search startup Exa hits $100M annualized revenue, growing ~10x a year for 3 years — TheIshanGoswami · 2026-10-14
- claude-as-cad: how to build real robots and parts without knowing CAD, by having Claude build its own CAD interface — _krishna_murthy · 2026-10-14
- Antigravity walkthrough for 500 devs at Paris Dev With AI: Agent Manager, IDE, CLI and SDKs — rseroter · 2026-10-14