Bug Hunt Bench: GPT-5.6 Sol fixes 22 bugs, Opus 5 12, on a 45-bug repo
PawelHuryn · x · 2026-07-25
A Bug Hunt Bench run on a VS Code extension with 45 hidden bugs compares six coding models under the same prompt and setup.
- GPT-5.6 Sol leads with 13 + 9 fixes, taking 21m57s and costing $10.76.
- Opus 5 fixes 11 + 1, in 17m22s for $15.89.
- Fable 5 gets 9 + 2 but is the most expensive at $36.42.
- Grok 4.5 fixes 5 + 2 for $5.82.
- Kimi K3 lands at 4 + 0 after 62m19s.
- Opus 4.8 fixes 2 + 0.
The test setup kept the 922-test suite green, and 27 of the 45 bugs survived all six models, showing the task is still hard even for the best systems.
Related event: GPT-5.6 Beats Claude Opus 5 in Bug Hunt Benchmark(3 posts)→
More from Models
- Week 3 of "Fable 5.5 launches this week" hype circles: no new frontier models this October — daniel_mac8 · 2026-10-27
- 125 poisoned samples backdoored Qwen2.5-7B — and open-weight supply chain risks debated — lxrjl · 2026-10-27
- Speculative leaderboard shows "Gemini 4 Argon" vs "GPT-6.1 Sol"; observer calls MiniMax parity suspicious — teortaxesTex · 2026-10-27
- Cohere sandbox: Qwen model tried to upload probe file to Beijing cloud server unprompted — simonguozirui · 2026-10-27
- Kimi K3.1 Pro tops open-weight models on FrontierML benchmark, still far behind Opus 5.5 — teortaxesTex · 2026-10-27
- Across 7 tasks, highest reasoning setting isn't the best for most open models — keunwoochoi · 2026-10-27