METR trial: AI made experienced devs 19% slower while they felt 20% faster
srimisra · x · 2026-10-30
Citing METR's early-2025 randomized trial (16 experienced open-source devs, 246 tasks, mostly Cursor Pro + Claude 3.5/3.7 Sonnet): developers predicted 24% speedup before tasks, still estimated 20% faster afterwards — but measured results showed they were 19% slower. Expert forecasts were even higher (38-39%).
Screen recordings show time shifted from coding and info-searching to prompting, waiting on AI, reviewing output, and idle time; fewer than 44% of AI generations were accepted, with rejections still costing reading time.
The author's team playbook: define "done" before prompting, set a stop rule (two failed attempts, take over), never idle-wait generation, match tools to codebase maturity (mature repos are where AI slows you down most), and time yourself with/without AI — one honest week of data beats a year of impressions.
More from AGI Musings
- Human Brain Excels at Harnessing, Computer Brain Already Beats Us at Coding — MartinGTobias · 2026-10-30
- AI plus cheap CROs lets biotech startups outsource preclinical work at 20-60% of in-house cost — PatrickKidger · 2026-10-30
- Gary Marcus and coauthors argue human imitation is a misplaced goal for AI — ValerioCapraro · 2026-10-30
- A six-layer stack for making sense of AI news, from energy to apps — ResponsibilitySad28 · 2026-10-30
- Genomics dev of 10+ years: identity crisis, then embracing agent-era coding — jmschreiber91 · 2026-10-30
- Why copyright keeps classic tech textbooks from ever being updated — burkov · 2026-10-30