Anthropic CTO Criticizes Offline Benchmarks, Calls for Real-World AI Metrics

Anthropic co-founder and CTO Joshua Saxe argues that single-dimension "capability" narratives mask the validity flaws of offline benchmarks, and that AI progress should instead be judged by productivity gains and real-world harm.

2026-11-09 ~ 2026-11-10 · 3 related posts

• Joshua Saxe: offline evals aren't progress — measure AI by labor productivity and real-world harm — typewriters · 2026-11-09
• AI researcher: single-dimension "capabilities" evals have poor predictive power — joshua_saxe · 2026-11-10

1 near-duplicate retellings: joshua_saxe