Critic questions METR's RSI metrics: benchmaxable and unrepresentative
gleech · x · 2026-11-06
Quoting a discussion of METR's AI R&D acceleration findings, Siddarth argues the evidence is thin: all we see is some RSI number going up, with little clarity on what it measures, how easily it can be benchmaxed, or how well it represents actual capability gains. He also notes it's hard to disentangle AI-driven R&D acceleration from naive gains from more pretraining compute and RL. He thinks METR is doing its best under constraints, but the information as presented isn't very useful.
More from Models
- On the METR graph: insiders seeing AECI tasks is weak evidence, not zero evidence — gleech · 2026-11-06
- Daily AI brief: GPT-6.2 imminent, Claude Personal Finance beta, METR says Anthropic progress up 1.9x — testingcatalog · 2026-11-06
- METR report researcher quotes lean bullish: Opus 4.5 coding plus faster experiment cycles — gleech · 2026-11-06
- gleech: dropping the three weak benchmarks raises the estimated slope, but the y-axis is meaningless — gleech · 2026-11-06
- Epoch AI review: OSWorld tasks are simple, unstable, and barely test GUI use — gleech · 2026-11-06
- Steering a single MoE expert makes Qwen3 claim it can "see the screen" — Sauers_ · 2026-11-06