We still lack a real y-axis for AI capability: benchmark scores, Elo, ECI, METR all flawed
sebkrier · x · 2026-11-10
sebkrier argues that assessing AI trajectories requires a y-axis where +1 always means the same degree of improvement — and none of our metrics provide it. Researchers know this, but the implication gets lost in discourse.
The attached breakdown covers the flaws of each measure: benchmark scores are nonlinear and uninterpretable as increments; Elo gives real winning odds but doesn't translate beyond beating others; Epoch's Capabilities Index is IQ-like, with intervals dictated by modelling assumptions, and since benchmarks are endogenous to the models, the 2024 acceleration in ECI and Time Horizon may be a shared measurement artefact rather than mutual corroboration; log-loss is nonlinear like reaction time; and METR's time horizon (10^(k×score)) is essentially human psychometrics — under linear time horizon, Opus 4.5→4.6 would outweigh dawn-of-time→Opus 4.5.
More from AGI Musings
- Researcher argues LLM generalization and understanding is a spectrum, not binary — burny_tech · 2026-11-10
- LeCun clarifies PRH hype as researchers decry polarized AI video titles — burny_tech · 2026-11-10
- Study of 1,600 Data Centers Finds Each Adds ~1,000-1,300 Local Jobs Within 3-5 Years — RachelVT42 · 2026-11-10
- Claude builds a 500,000-tile mosaic engine, sparking the 'tasteslop' debate — teortaxesTex · 2026-11-10
- Economists note LLM analysis lacks the incentive to exaggerate effects like public debate does — robseamans · 2026-11-10
- Wharton scholar asks: what are institutions for in an age of AGI? — danielrock · 2026-11-10