Study: 87% of measured AI gains came by 2022, labs cherry-pick evals
gerardsans · x · 2026-10-16
Gerard Sans cites a study of best-available scores across 25 practical task types (79 measures) by model release year to argue against exponential AI progress: 87% of the measured gain was already achieved by 2022, with only 12 points added in the four years since (0.87 with GPT-3.5 to 0.99 projected for 2026's 19 new models).
He calls RSI an "empty term" — there's no recursion, just iteration, where some dimensions improve while others plateau, like drafting an email over 2-3 iterations with no reason for indefinite improvement. He also accuses labs of exaggerating progress by cherry-picking internal evals that match their post-training regimes instead of measuring real-world user performance.
Related event: RSI Criticized as Empty Term as Data Shows Gains Predating 2022(2 posts)→
More from AGI Musings
- Kevin Roose joins Big Technology Podcast to discuss the bumpy road to AGI — kevinroose · 2026-10-16
- Terence Tao: math is leaving its romantic 'age of heroes' for an industrial era — haider1 · 2026-10-16
- Stripe's Katie Dill AI design talk becomes Lenny's third most-viewed video — lennysan · 2026-10-16
- zetalyrae mocks rationalists: 'poverty exists, therefore perpetual AI totalitarianism' — zetalyrae · 2026-10-16
- Jon Stokes: alignment is impossible because utility functions are hindsight constructions — sebkrier · 2026-10-16
- Philosopher Carissa Véliz: a free society needs preparation, not AI oracles — CarissaVeliz · 2026-10-16