Benchmarking AI Coding Agents: o3 Took 3 Hours with 10 Mistakes, GPT-5.5 Wrote Empty Scaffolding

AI Engineer · youtube · 2026-08-09

Denys Linkov from Wisedocs shared an honest audit of using AI coding agents to refactor a massive medical claims codebase.

Model Performance: For a single refactor task, o3 took 3 hours in Cursor and still shipped 10 major mistakes. Opus 4.8 essentially got it in one pass at 1/5th the effort. GPT-5.5 wrote 2,000 lines in 10 minutes, but it was just empty scaffolding with missing models.
Task Success Rates: Linkov suggests reading the METR task length curve at 80-90% success, as launching an hour-long agent run on coin-flip odds usually just wastes time.
Verdict: Doing the refactor beat deferring it. Commit velocity rose significantly, work that took months now ships in under a week, and developers volunteer across the repo outside their core areas.

More from coding & agent