Measuring AI's impact: focus on final outputs, not lines of code
soumitrashukla9 · x · 2026-09-21
- testingham argues AI impact measurement should prioritize aggregate final outputs (e.g., cyber exploits discovered, math problems solved, algorithmic efficiency) over inputs (time, token spend) and intermediate proxies (papers, LOC, commits, self-reported speedups), since proxies can move without outputs changing.
- Rishi Bommasani adds we need more measurement initiatives deeply immersed in frontier AI; most downstream measurement orgs don't realize upstream AI adoption will change their work.
- Funders are backing efforts like RCTs on firm-level productivity effects, but growth in this space is too slow; methodologically, solve attribution as simply as possible.
Related event: Study: AI Sharply Accelerates Cybersecurity Discoveries but Little Else(2 posts)→
More from AGI Musings
- Greenblatt: Most senior capability researchers at OpenAI and Anthropic see AI takeover risk as significant — RyanGreenblatt · 2026-09-21
- Quant Researchers Are Quitting $600K Jobs for AI Safety Roles at a Crazy Pace — himanshustwts · 2026-09-21
- AI Could Lift European Productivity 1% in Five Years, But Value May Flow Abroad — TansuYegen · 2026-09-21
- Founder: Organizations Full of AI Users Should Cut the Middlemen, Not Keep Human Loops — bindureddy · 2026-09-21
- "Everyone wants to slow down, as long as nobody else speeds up": on Anthropic's AI race paradox — TansuYegen · 2026-09-21
- The irony of AI: humans now personalize writing to avoid looking like machines — SuB8u · 2026-09-21