Benchmark Error Bars Too Wide: Data Underpowered to Prove Recent AI Speedup
tobyordoxford · x · 2026-11-06
tobyordoxford discusses why recent measurement error bars in AI capability trends are so much larger than earlier ones, attributing it to (1) a lack of hard enough benchmarks and (2) too few models at similar capability levels to calibrate benchmark difficulty. He notes that a speedup of 1.0x still fits within the error bars, meaning the data is underpowered to establish any recent acceleration from recursive self-improvement—he guesses there is one, but it's hard to tell from current data.
Related event: Oxford philosopher explains why AI benchmark error bars have widened(2 posts)→
More from Research
- Project AURORA launches at Harvard: AI cancer co-scientists backed by CRUK and NCI — zakkohane · 2026-11-06
- Google DeepMind Open-Endedness team hiring Student Researchers in Toronto — robertarail · 2026-11-06
- VisMatch v1.4.0 adds RaCo, UPAL, REDI-Match: 50+ image matchers in 5 lines — rsasaki0109 · 2026-11-06
- Trainable [DUMP] Prefix Tokens Reduce Representational Interference in Retrieval Embeddings — nlpai-lab · 2026-11-06
- Nace AI open-sources Drex decision models: 27B rivals frontier LLMs at 183ms — omarsar0 · 2026-11-06
- MIT's David Bau Challenges Gates, Hinton, Bengio: Show Me One Unthinkable Thought — davidbau · 2026-11-06