Multi-Model Bug Hunting Eval: Who Fixes Code Best
PawelHuryn · x · 2026-07-19
The author planted 45 bugs in their VS Code extension repo grok-build and tested five models under their respective native harnesses, using the same prompt and high effort conditions for bug finding and fixing.
The results show:
- GPT-5.6 Sol found 13 hidden bugs, plus 9 pre-existing real bugs in the repo, and added 7 new tests; the author considers it a "bug-tracking maniac" with the strongest execution.
- Fable 5 had the sharpest diagnostics but was the most expensive, costing about $36; better suited for judgment calls rather than direct execution.
- Grok 4.5 offered the best cost-performance ratio, costing about $6 and running fast; it even spawned sub-agents unprompted, making it ideal for routine work.
- Kimi K3 was very slow, taking 62 minutes to fix 4 bugs, but fixed them all correctly, making it suitable for overnight runs.
- Opus 4.8 only confirmed and reported what it could fully verify, making it quite conservative.
The author's recommended combo is: Fable 5 for judging/reviewing, and GPT-5.6 Sol for execution; if cost-effectiveness is the priority, run Grok 4.5 solo, and use Fable or Sol for backup when stuck. Ultimately, 27/45 bugs escaped all models.
More from coding & agent
- Astra storyboards plus Minimax H3 per-shot generation boost video success rates — Hailuo_AI · 2026-09-11
- Codex tip: use Sol with Astra and Luna sub-agents to save usage — pvncher · 2026-09-11
- agents-best-practices: a provider-neutral Agent Skill for designing and auditing agentic harnesses — tom_doerr · 2026-09-11
- Cognition's SWE-2 uses a KKT duality argument in RL to shift the effort Pareto curve — YouJiacheng · 2026-09-11
- First-ever Three.js Conference lands in Paris, with a panel on AI-shortened design workflows — OdinLovis · 2026-09-11
- Agile co-author Ron Jeffries publishes 'Resist AI', urging developers to push back — mborch · 2026-09-11