Bug Hunt Bench: GPT-5.6 Sol fixes 22 bugs, Opus 5 12, on a 45-bug repo
PawelHuryn · x · 2026-07-25
A Bug Hunt Bench run on a VS Code extension with 45 hidden bugs compares six coding models under the same prompt and setup.
- GPT-5.6 Sol leads with 13 + 9 fixes, taking 21m57s and costing $10.76.
- Opus 5 fixes 11 + 1, in 17m22s for $15.89.
- Fable 5 gets 9 + 2 but is the most expensive at $36.42.
- Grok 4.5 fixes 5 + 2 for $5.82.
- Kimi K3 lands at 4 + 0 after 62m19s.
- Opus 4.8 fixes 2 + 0.
The test setup kept the 922-test suite green, and 27 of the 45 bugs survived all six models, showing the task is still hard even for the best systems.
More from Models
- Anthropic’s latest chart crime looks like over-trusting Claude, not deliberate hype — herbiebradley · 2026-07-25
- Claude Opus 5 fixes 11 of 45 hidden bugs, versus 2 for Opus 4.8 — PawelHuryn · 2026-07-25
- Miles Brundage says Anthropic’s chart issue is really about over-trusting Claude — Miles_Brundage · 2026-07-25
- A benchmark chart becomes an AI meme after viewers spot the messy numbers — Miles_Brundage · 2026-07-25
- FrontierCode 1.1 shows Opus 5 can score lower under stricter reasoning settings — andrew_n_carr · 2026-07-25
- Claude Opus 5 builds a Rocket League clone on just 27% of a Max plan — soumitrashukla9 · 2026-07-25