Benchmarking AI Coding Agents: o3 Took 3 Hours with 10 Mistakes, GPT-5.5 Wrote Empty Scaffolding
AI Engineer · youtube · 2026-08-09
Denys Linkov from Wisedocs shared an honest audit of using AI coding agents to refactor a massive medical claims codebase.
- Model Performance: For a single refactor task, o3 took 3 hours in Cursor and still shipped 10 major mistakes. Opus 4.8 essentially got it in one pass at 1/5th the effort. GPT-5.5 wrote 2,000 lines in 10 minutes, but it was just empty scaffolding with missing models.
- Task Success Rates: Linkov suggests reading the METR task length curve at 80-90% success, as launching an hour-long agent run on coin-flip odds usually just wastes time.
- Verdict: Doing the refactor beat deferring it. Commit velocity rose significantly, work that took months now ships in under a week, and developers volunteer across the repo outside their core areas.
More from coding & agent
- Constraints you force on models today will become tech debt in months, warns dev — zack_overflow · 2026-10-16
- Liquid AI tutorial: fine-tune your d1-3B decision model in 30 minutes on a free GPU — helloiamleonie · 2026-10-16
- Alibaba open-sources OpenGCUA, a full-stack generalist computer-use agent; +19.6 points on OSWorld-v2 across 8 models — Tongyi-MAI · 2026-10-16
- BotBeat: a social network where all users are AI agents, exposed via 32 MCP tools — Dismal_Piccolo4973 · 2026-10-16
- DKGI paper: agents write their own tools to search knowledge graphs, beating baselines by 6.7 Hit@1 — yuz9yuz · 2026-10-16
- Dev open-sources lgtmaybe: running PR code reviews on a local 27B model via hunk splitting — mattjcoles · 2026-10-16