On METR's new graph: leaked AECI benchmarks are the usual weaksauce, but insiders are freaked out
gleech · x · 2026-11-06
gleech argues no one can react appropriately to METR's new trend graph since we lack evidence to evaluate ourselves. But the report leaking three AECI benchmarks shows they're the usual weaksauce — poor external validity, public test sets, label noise (MMMLU is just MMLU again). He calls the reassurance effect remarkable, and notes insiders being freaked out by AECI components is weak evidence, not zero evidence.
Related event: METR's New Capability Chart Sparks Debate Over Leaked AECI Benchmarks(2 posts)→
More from AGI Musings
- Friday's jobs report forecast: just 55,000 new jobs as AI starts showing up in the numbers — rand_longevity · 2026-11-06
- Gary Marcus: AI isn't over — what may end is insane valuations and data center spending — GaryMarcus · 2026-11-06
- IIT Goa professor redesigns exam: students write their own questions, then answer them — kingslyj · 2026-11-06
- "We had the Transformer paper before LLMs — nothing even close since," dev argues — dosco · 2026-11-06
- Toby Ord: jump-plus-steeper-slope mixed hypothesis beats a steeper slope alone — tobyordoxford · 2026-11-06
- Readback Labs proposes mandatory audits requiring AI execs to prove they understand AI R&D — ronbodkin · 2026-11-06