gleech: dropping the three weak benchmarks raises the estimated slope, but the y-axis is meaningless
gleech · x · 2026-11-06
gleech reiterates his critique of METR's AECI benchmark basket: he's not claiming three weak benchmarks explain away the nominal acceleration (removing them increases the estimated slope), but inferring the other AECI components won't be good either, making the y-axis uninformative.
Related event: Researchers Debate METR Acceleration Report, Urge Reading the Source(3 posts)→
More from Models
- Custom GPTs die Dec 11: if you use someone else's GPT, you can't migrate it — Ok_Negotiation_2587 · 2026-11-06
- Thinking Machines Ships Inkling 2 Small: 68-Hour Pretrain, Top Non-China Open-Weights Model — Latent Space · 2026-11-06
- Kimi K3.1 and Tencent Hy4 accused of gutting cybersecurity skills from open-weight models — Fresh_Sun_1017 · 2026-11-06
- On the METR graph: insiders seeing AECI tasks is weak evidence, not zero evidence — gleech · 2026-11-06
- Daily AI brief: GPT-6.2 imminent, Claude Personal Finance beta, METR says Anthropic progress up 1.9x — testingcatalog · 2026-11-06
- METR report researcher quotes lean bullish: Opus 4.5 coding plus faster experiment cycles — gleech · 2026-11-06