Toby Ord on why benchmark error bars have grown so large in recent AI evaluations
tobyordoxford · x · 2026-11-06
Oxford philosopher Toby Ord discusses why error bars in AI capability evaluations have become much larger recently, offering two explanations:
- Lack of hard-enough benchmarks
- Lack of many other models at similar capability levels to calibrate benchmark difficulty
He notes METR's report covers this in depth, though he hasn't fully absorbed it yet.
Related event: Oxford philosopher explains why AI benchmark error bars have widened(2 posts)→
More from Models
- Bittensor miners train open-source decision model REAX 1.3 — const_reborn · 2026-11-06
- Bittensor subnet 92 releases REAX 1.3, an open-source decision model blending miner models — const_reborn · 2026-11-06
- Kimi K3.1 Pro scores 9.9% on agentic WeirdML v3, ahead of Kimi-K3 but behind Claude Haiku 5.5 — scaling01 · 2026-11-06
- Rumor: GLM-5.5 to come in 32B and 122B sizes, community figure says no — TheZachMueller · 2026-11-06
- User screenshot suggests Anthropic's weekly Claude limits feel effectively unlimited — StewartalsopIII · 2026-11-06
- Leonie Shares Full Slide Deck on Building Decision Models, Still Early Days — helloiamleonie · 2026-11-06