Oxford philosopher explains why AI benchmark error bars have widened
Oxford philosopher Toby Ord explains why error bars in AI capability evaluations have widened recently, citing a lack of sufficiently hard benchmarks and comparable models for calibration, which leaves evidence for AI speeding up research inconclusive.
2026-11-06 ~ 2026-11-06 · 2 related posts
- Episode 1: Anthropic Says It May Imminently Hit RSP Threshold for AI-Accelerated R&D(2026-11-03, 7 posts)
- Episode 2: AI 2027 Authors Warn Anthropic's 2x Threshold Could Accelerate Superintelligence Timeline(2026-11-05, 2 posts)
- Episode 3: METR's First Inside-Data Assessment Finds AI Has Roughly Doubled Anthropic's R&D Pace(2026-11-06, 65 posts)
- Episode 4: Anthropic's Internal Models Reportedly Surpass Mythos 5(2026-11-06, 2 posts)
- Episode 5: METR report on Anthropic AI R&D acceleration heavily redacted, sparking transparency dispute(2026-11-06, 11 posts)
- Episode 6: Report: Anthropic's Next Model Finished Training, Tops All METR Benchmarks(2026-11-06, 2 posts)
- Episode 7: Insider: 7 of Anthropic's Last 8 Frontier Models Never Publicly Benchmarked(2026-11-06, 2 posts)
- Episode 8: METR Says Anthropic Canceled Compute Assessment Pilot, Sparking Safety Criticism(2026-11-06, 2 posts)
- Episode 9: AI Safety Community Criticizes Anthropic's Acceleration Strategy(2026-11-06, 2 posts)
- Episode 10: Anthropic Co-founder Responds to METR Report, Hints at Internal Acceleration Findings(2026-11-06, 2 posts)
- Episode 11: Oxford's Toby Ord Pushes Back on METR's 2x R&D Speedup: Jump-Plus-Slope Fit Suggests Only 1.4x(2026-11-06, 9 posts)
- Episode 12: Oxford philosopher explains why AI benchmark error bars have widened(2026-11-06, 2 posts)
- Benchmark Error Bars Too Wide: Data Underpowered to Prove Recent AI Speedup — tobyordoxford · 2026-11-06
- Toby Ord on why benchmark error bars have grown so large in recent AI evaluations — tobyordoxford · 2026-11-06