GPT-5.6 Sol fixes 31 of 105 hidden bugs in a two-repo benchmark
PawelHuryn · x · 2026-07-27
What the benchmark measured
- The author ran 7 frontier models on 105 hidden bugs across two unrelated codebases: a VS Code extension and an LMS.
- Each model got one round per repo, with strict diff grading: a fix counted only if the bug was actually removed.
- The benchmark used ground-truth diffs, blind judges, anonymized submissions, and a withheld answer key.
Key results
- GPT-5.6 Sol: 31 bugs fixed total
- Fable 5: 24
- Opus 5: 21
- Kimi K3: 21
- Grok 4.5: 16
- Opus 4.8: 9
- Sonnet 5: 9
Notable takeaways
- In repo 1, 45 bugs were planted; 28 survived every model.
- In repo 2, 60 shipped regressions were used; 33 survived every model.
- The author argues that a green test suite does not prove a codebase is clean, and a model’s confident “no issues found” report can still miss many real bugs.
- The chart also shows large differences in wall-clock time and cost across models.
Related event: Frontier Models Miss Over Half of 105 Hidden Bugs in Coding Benchmark(4 posts)→
More from Models
- Google’s Gemini 3.6 Flash could win by matching Sonnet quality at a lower cost — haider1 · 2026-07-27
- Users Report Severe Downgrade in Opus 5: Hallucinations and Math Errors — whatsallthiss · 2026-07-27
- A Reddit explainer breaks down MoE, KV cache, MLA and KDA behind Kimi K3 — MohamedKadri_ · 2026-07-27
- Reddit asks which local model works best for coding, planning and VS Code workflows — naunen · 2026-07-27
- Opus 5's ARC-AGI-3 Leap Debunked as 'Complete Slop' Due to Benchmark Design Flaws — scaling01 · 2026-07-27
- MineBench says Opus 5.0 beats Fable 5 on build quality but costs 64% more — ENT_Alam · 2026-07-27