Measuring AI's impact: focus on final outputs, not lines of code
soumitrashukla9 · x · 2026-09-21
- testingham argues AI impact measurement should prioritize aggregate final outputs (e.g., cyber exploits discovered, math problems solved, algorithmic efficiency) over inputs (time, token spend) and intermediate proxies (papers, LOC, commits, self-reported speedups), since proxies can move without outputs changing.
- Rishi Bommasani adds we need more measurement initiatives deeply immersed in frontier AI; most downstream measurement orgs don't realize upstream AI adoption will change their work.
- Funders are backing efforts like RCTs on firm-level productivity effects, but growth in this space is too slow; methodologically, solve attribution as simply as possible.
Related event: Study: AI Sharply Accelerates Cybersecurity Discoveries but Little Else(2 posts)→
More from AGI Musings
- Agents' bottleneck isn't intelligence but goal commitment — and that's good for alignment — mimi10v3 · 2026-11-12
- Google Cloud survey: 78% say legacy enterprise vendors may not lead the AI era — rseroter · 2026-11-12
- Safety researchers blast OpenAI/Anthropic plan: automating alignment research with possibly misaligned AI — trevposts · 2026-11-12
- NYT essay: resisting Big Tech and AI takes collective action, not willpower — TuhinChakr · 2026-11-12
- Ethan Mollick: 'talk to your documents' RAG solutions already feel dated — emollick · 2026-11-12
- Anthropic's pitch hasn't changed in 5 years: 'AI is winner-take-all, we will build god' — menhguin · 2026-11-12