GPT-6 Sol Tanks Hands-on Bug Hunt: Fixes 29 of 105 Hidden Bugs, Cheapest at $9.93
PawelHuryn · x · 2026-09-23
The author tested frontier models on two real repos containing 105 hidden bugs (find and fix what you can), and GPT-6 Sol showed a big regression:
- GPT-6 Astra (max): 45
- GPT-5.6 Sol (max): 43.5
- Opus 5.5 (max): 41.7
- Muse Spark 1.3 (max): 32.2
- GPT-6 Sol (max): 29.3, last place
Cost flips the picture when normalized to API-equivalent pricing: GPT-6 Sol costs just $9.93 vs GPT-5.6 Sol's $95.25; GPT-6 Astra $33.04, Opus 5.5 $58.53, Muse Spark 1.3 $18.11. More effort levels (xhigh/high/medium/low) are being posted in the thread.
Related event: Bug Hunt Bench: GPT-6 Astra Tops 105-Bug Fixing Test, MiMo Stuns on Value(18 posts)→
More from Models
- Theory: Typos Disrupt LLM Tokenization, Which May Explain Why AI Text Has a Distinct Smell — menhguin · 2026-11-11
- LMArena co-founder explains AI evaluation methodology and policy in new podcast — MattPerault · 2026-11-11
- Runway open-sources Praxis-1, turning video pretraining into real robot control — runwayml · 2026-11-11
- DeepMind launches AlphaGenome 2, spotting nearly 2x more gene-regulating mutations — GoogleDeepMind · 2026-11-11
- MazeBench: Fable 5.2 scores 15%, near-frontier at half price; Gemini 4 Carbon 11% — patience_cave · 2026-11-11
- ERNIE 5.5 Flash Preview enters LMArena Battle Mode for crowdsourced ranking — arena · 2026-11-11