Multi-Model Bug Hunting Eval: Who Fixes Code Best
PawelHuryn · x · 2026-07-19
The author planted 45 bugs in their VS Code extension repo `grok-build` and tested five models under **their respective native harnesses**, using the same prompt and **high effort** conditions for bug finding and fixing. The results show: - **GPT-5.6 Sol** found 13 hidden bugs, plus 9 pre-existing real bugs in the repo, and added 7 new tests; the author considers it a "bug-tracking maniac" with the strongest execution. - **Fable 5** had the sharpest diagnostics but was the most expensive, costing about **$36**; better suited for judgment calls rather than direct execution. - **Grok 4.5** offered the best cost-performance ratio, costing about **$6** and running fast; it even spawned sub-agents unprompted, making it ideal for routine work. - **Kimi K3** was very slow, taking **62 minutes** to fix 4 bugs, but fixed them all correctly, making it suitable for overnight runs. - **Opus 4.8** only confirmed and reported what it could fully verify, making it quite conservative. The author's recommended combo is: **Fable 5 for judging/reviewing, and GPT-5.6 Sol for execution**; if cost-effectiveness is the priority, run **Grok 4.5** solo, and use Fable or Sol for backup when stuck. Ultimately, **27/45** bugs escaped all models.
More from coding & agent
- Freerange adds static numeric-range proofs for UI code without running the app — cnakazawa · 2026-07-21
- pen.dev lets developers compare frontier models in parallel on frontend design tasks — yakuzeg · 2026-07-21
- ComfyUI slowdown may come from models reloading from disk instead of RAM — J6j6 · 2026-07-21
- ZooData’s API takes a URL and returns structured data in the same request — dr_cintas · 2026-07-21
- ZooData turns any web page into structured JSON for agents with one API call — dr_cintas · 2026-07-21
- Side-by-side model test lands both answers on the first try, then shifts to loop engineering — glenbeer · 2026-07-21