Multi-Model Bug Hunting Eval: Who Fixes Code Best
PawelHuryn · x · 2026-07-19
The author planted 45 bugs in their VS Code extension repo grok-build and tested five models under their respective native harnesses, using the same prompt and high effort conditions for bug finding and fixing.
The results show:
- GPT-5.6 Sol found 13 hidden bugs, plus 9 pre-existing real bugs in the repo, and added 7 new tests; the author considers it a "bug-tracking maniac" with the strongest execution.
- Fable 5 had the sharpest diagnostics but was the most expensive, costing about $36; better suited for judgment calls rather than direct execution.
- Grok 4.5 offered the best cost-performance ratio, costing about $6 and running fast; it even spawned sub-agents unprompted, making it ideal for routine work.
- Kimi K3 was very slow, taking 62 minutes to fix 4 bugs, but fixed them all correctly, making it suitable for overnight runs.
- Opus 4.8 only confirmed and reported what it could fully verify, making it quite conservative.
The author's recommended combo is: Fable 5 for judging/reviewing, and GPT-5.6 Sol for execution; if cost-effectiveness is the priority, run Grok 4.5 solo, and use Fable or Sol for backup when stuck. Ultimately, 27/45 bugs escaped all models.
More from coding & agent
- Spring AI 2.1.0-RC1 adds tool-call confirmation, pausing agent loops before real-data access — therealdanvega · 2026-10-26
- 250 Builders Hack With Codex at Berlin Community Hackathon Ahead of OpenAI DevDay — paw_lean · 2026-10-26
- Building AI bots on raw APIs? Devs say models are 'dumb without the harness' — madeofAjala · 2026-10-26
- HarnessMM cuts agent tokens per task by up to 88% while raising success rates — omarsar0 · 2026-10-26
- Microsoft demos intent-classifying agents built with Copilot App and Foundry, deployable across Teams and Outlook — marlene_zw · 2026-10-26
- Looking for a wearable mic with best battery life that feeds recordings to personal agents via MCP — altryne · 2026-10-26