Palisade's Jeff Ladish: skip the 40 benchmarks, ask if agents finish real work
FinanceYF5 · x · 2026-11-06
Palisade Research's Jeff Ladish argues AI capability measurement is broken: instead of 40 benchmarks, the question should be whether agents can autonomously complete economically valuable work.
- His biggest capability jump came around Anthropic's Fable model (released June): "Before and after Fable, the difference was huge for me. I went from using agents sometimes to using them every day."
- Past evaluations of long-horizon tasks relied on METR hiring software engineers, not benchmark scores.
- What he wants to see: "Take us inside Anthropic — what's it like to be an Anthropic engineer? How many agents do they use daily?" He calls for a return to evaluating AI on real work it can finish autonomously.
Related event: Palisade researcher: judge AI by real work, not 40 benchmarks(2 posts)→
More from AGI Musings
- RSI isn't about acceleration—it's removing humans as the bottleneck — menhguin · 2026-11-06
- Stop Measuring AI in Gigawatts and GPUs, Architect David Linthicum Argues — DavidLinthicum · 2026-11-06
- METR Reviews Anthropic's Automated R&D Risk Claims With Access to Nonpublic Report — DKokotajlo · 2026-11-06
- Forecast model: 2x progress rate in Dec 2026 implies ~4.5x capability uplift — arthurcolle · 2026-11-06
- Balaji taunts Anthropic and OpenAI: you fear-mongered, now find out — beffjezos · 2026-11-06
- David Patterson estimates economic output will grow 25% next year — davidpattersonx · 2026-11-06