Debate over METR benchmark fit: whether the 1.9x slope increase holds up
JacquesThibs · x · 2026-11-08
Jacques Thibs and AI safety researcher Jeff Ladish debate the controversy around METR's AECI benchmark. Ladish thinks Toby (Ord) is wrong but is glad he's arguing his position; Thibs pushes back, arguing that "METR has other information we don't have" isn't convincing for the specific fit of the AECI benchmark — the additional information is far more relevant to the overall 2x speedup number than the 1.9x slope increase itself.
A concrete methodological dispute in AI capability extrapolation circles.
Related event: Researchers Debate Reliability of METR 2x Speedup Claim(2 posts)→
More from Research
- MuJoCo 4.0 ships with speculative contacts, permanent soft-body deformation and MuJoCo Studio — ZoubinGhahrama1 · 2026-11-08
- mitsuhiko: djs' JS "heap dumps" may beat real dumps for durable coding agents — mitsuhiko · 2026-11-08
- Salesforce paper: RL-trained experts merged via OPD save 42% FLOPs vs external-teacher distillation — rohanpaul_ai · 2026-11-08
- Albert Gu: Bitter Lesson isn't 'just scale Transformers' — algorithmic innovation matters — sewon__min · 2026-11-08
- Autoware open-sources an off-road driving simulator built on NVIDIA Isaac Sim — rsasaki0109 · 2026-11-08
- The meta-question as agent swarms scale: can models beat gradient descent at its own game? — JensHonack · 2026-11-08