I spent $200 benchmarking AI bug-fixing: Opus tops at 94%, DeepSeek matches for 1/26 the cost
narrow-adventure · reddit · 2026-10-23
To settle whether CLI+skills or MCP is the better way to harness AI for bug-fixing, the author built a 62-case benchmark and spent $200 running it. Claude Opus ranked first at 94% fix rate; DeepSeek 4.1 flash hit 87% at $2 vs Opus's $52 — 'a year ago, 84% of issues fixed automatically for $2 would have been crazy.' The takeaway: open-weight models are closing in on frontier models at a fraction of the price. Next steps: multiple runs, better skills, a final MCP vs skills+CLI ranking, and more open models. The benchmark is public at tracewayapp.com/benchmarks; the author discloses they maintain Traceway (MIT) but says the bench was a side quest, not a pitch.
More from coding & agent
- rez0 pushes back on 'prompt engineering is dead': prompts still make a world of difference — rez0__ · 2026-10-23
- Banks keep piloting agents, but the control plane is what gets missed — krishnan · 2026-10-23
- Agent swarms collaborate 10K hours in simulated GitHubs: Gemini-3.8-flash tops, gpt-6-astra worst — RulinShao · 2026-10-23
- LiteLLM v1.107.0 supply-chain compromise: don't pip fix, delete cook.pth and rotate keys — alvelda · 2026-10-23
- Meta trains an 8B model to pick tools better than a 402B Llama-4-Maverick — dair_ai · 2026-10-23
- Coding agents declaring false 'Done' — a Reddit experiment adds an independent controller to verify completion — aldi949 · 2026-10-23