105 planted bugs tested: GPT-6 Astra tops at 45, GPT-6 Sol shows big degradation
JohnMcKeownn · x · 2026-09-24
Pawel Huryn ran frontier coding models against 105 hidden bugs in two real repos: GPT-6 Astra (max) fixed 45, GPT-5.6 Sol 43.5, Opus 5.5 41.7, Muse Spark 1.3 32.2, and GPT-6 Sol only 29.3—a large degradation. API-equivalent cost varies widely too: GPT-6 Astra $33, Opus 5.5 $58.5, GPT-5.6 Sol over $95. The bug list stays private to keep the benchmark valid, but test coverage is published.
Related event: GPT-6 Astra Tops Bug Hunt Bench While Sol Version Regresses(2 posts)→
More from coding & agent
- The decade of agents: simulate real customers to test your agent before launch — EXM7777 · 2026-11-11
- FluentCRM MCP Server lets you manage marketing automation from Cursor or Claude — modelcontextprotocol · 2026-11-11
- Databricks' revised Genie agent tells CEO Ali Ghodsi the unvarnished truth — TechCrunch AI · 2026-11-11
- Elvis Saravia: agents will subsume traditional software — and can even cut code bloat — omarsar0 · 2026-11-11
- Anthropic's Fable 5.2 built a professional video editor overnight, unsupervised — every · 2026-11-11
- Using a third-party inference provider with your coding harness hands it shell access to your machine — TheZachMueller · 2026-11-11