Palisade's Jeff Ladish calls to revive METR time-horizon estimates for new models
JeffLadish · x · 2026-11-06
Palisade Research's Jeff Ladish argues the real test of AI progress is whether agents can actually do the work, not 40 benchmarks. He recalls when METR measured capabilities by having software engineers tackle long-horizon tasks, and says the biggest jump he felt was between pre- and post-Fable models — going from occasionally using agents to using them daily.
He hopes METR or others will bring back the METR graph estimating task time horizons for new models: weeks-long tasks are hard to measure, but surely several-week horizons can be quantified.
More from Models
- NaceAI open-sources Drex decision models, hitting SOTA 67.52 on Decision Index — nischay_twt · 2026-11-06
- Users say Claude Opus 4.8 is 'so stubborn even its own classifiers can't stop it' — repligate · 2026-11-06
- Miles Brundage: Inkling 2 Small is good — Miles_Brundage · 2026-11-06
- Users claim Claude performs better with an empty system prompt than Anthropic's official one — repligate · 2026-11-06
- Palisade's Jeff Ladish: models better at everything but still not funny or persuasive — JeffLadish · 2026-11-06
- Leaker DM claims Qwen4 is dropping, but even the timeline is a guessing game — bclavie · 2026-11-06