105 hidden bugs, 5 flagship models: GPT-6 Sol lands last while GPT-5.6 Sol costs $95 per run
ronnhue · x · 2026-09-23
PawelHuryn ran a real-work benchmark across two repos containing 105 hidden bugs (find and fix as many as possible):
- GPT-6 Astra (max): 45
- GPT-5.6 Sol (max): 43.5
- Opus 5.5 (max): 41.7
- Muse Spark 1.3 (max): 32.2
- GPT-6 Sol (max): 29.3 — a major degradation
API-equivalent costs flip the picture further: GPT-6 Astra $33, GPT-5.6 Sol a steep $95.25, Opus 5.5 $58.53. Commenters note the luna series feels strong for its cost.
Takeaway: GPT-6 Sol underperforms badly on real coding work, and the priciest option isn't necessarily the most cost-effective.
Related event: Bug Hunt Bench: GPT-6 Astra Tops 105-Bug Fixing Test, MiMo Stuns on Value(18 posts)→
More from coding & agent
- As models improve, writing good evals gets harder and more important — amankhan · 2026-11-12
- Omarchy 4.6 lets you share local AI models across your tailnet, says DHH — alexcovo_eth · 2026-11-12
- Coding harnesses hand shell access to inference providers — 4-step gateway mitigation — shensi · 2026-11-12
- Slack's David Hoang: agents beat people on coordination, proposes 'boundary agents' — davidhoang · 2026-11-12
- CS student mourns coding flow states he'll never experience in the AI era — chrissyykat · 2026-11-12
- AI Engineer Code Summit features its first all-female keynote lineup — altryne · 2026-11-12