Joshua Saxe: offline evals aren't progress — measure AI by labor productivity and real-world harm
typewriters · x · 2026-11-09
Anthropic co-founder and CTO Joshua Saxe praised an investment-normalized capabilities growth chart for grounding AI progress in capital markets, but argued the discourse must go further: stop treating roll-ups of offline evals as the true objects of progress and risk.
He contends AI has been around long enough that the real world is the object: on capabilities, the real metrics are AI's causal impacts on labor productivity, GDP growth, and self-reported happiness; on risk, real cyber damages, real bio near-misses, and real misalignment harms.
Offline evals remain useful, he says, but the field needs a culture where — when evals are equated to "capabilities" — they are explicitly assessed for their predictive power relative to these real-world outcomes.
More from AGI Musings
- Cambridge AI safety prof shares kid posting 'March for an AI Treaty' flyer — DavidSKrueger · 2026-11-09
- Public GitHub pushes double in one year to 408M per quarter amid AI-generated code surge — DimitrisPapail · 2026-11-09
- Companies handed out AI without saying what they expected — so workers just burned out — feregri_no · 2026-11-09
- Veteran pentester: AI agent swarms get blocked fast — breaches still come from bad passwords — ziv_ravid · 2026-11-09
- Silicon Valley warns AI will kill everyone — normies respond by voting against data centers — DavidSKrueger · 2026-11-09
- Banning AI Is Like Banning Uranium Enrichment, Safety Researchers Quip — DavidSKrueger · 2026-11-09