76% of orgs ship AI coding assistants but only 34% can measure impact — activity metrics are misleading
rseroter · x · 2026-09-19
rseroter engages with Bharat Sharma's article arguing AI hasn't made engineering metrics obsolete — it made activity proxies (lines of code, commits, PR counts) actively misleading.
Key evidence:
- A 2026 Halkwinds survey of 758 engineering orgs: 76% rolled out an AI coding assistant org-wide (up from 41% in 2024), but only 34% could point to measurable, audited delivery changes.
- METR's 2025 RCT: 16 experienced open-source devs on 246 tasks were 19% slower with AI — while believing they'd be faster.
- Randomized field experiments at Microsoft, Accenture, and a Fortune 100 company (4,867 devs) estimated 26.08% more completed tasks; an observational study of 16,223 devs showed up to +40.5% PRs among heavy Copilot users (self-selected).
Sharma's prescription: measure system-level outcomes instead, with a 90-day plan; activity metrics still tell you something in context, the mistake is treating them as primary indicators.
More from coding & agent
- Xcode 27.1 Adds iPhone Duo Readiness Skill Auditing Seven Classes of App Compatibility Issues — rudrank · 2026-09-19
- Law professor's five-week method turns students from prompters into AI tool builders — jkubicki · 2026-09-19
- opencode go at $10/month offers ~4x the tokens of going direct via OpenRouter — craigbalding · 2026-09-19
- No Prompt, No JSON Parsing: Jev's Typed API Maps Straight onto Java Records — therealdanvega · 2026-09-19
- Non-technical user rebuilds a community bot with Codex in ~20 minutes — Travi3000 · 2026-09-19
- Agent categorizes 63,045 emails in under 3 minutes for less than $1 — NathanWilbanks_ · 2026-09-19