Ethan Mollick: benchmarks miss long-horizon work; small models need orchestration
emollick · x · 2026-10-16
Ethan Mollick argues that nearly all benchmarks test single-answer, short-horizon tasks, while much economically valuable work consists of complex long-horizon tasks where "good enough" intelligence may not be enough.
Saving money with small models in the short term can cost you in work quality: small models can do plenty of valuable work, but for many tasks they should be orchestrated and error-checked by larger ones. He links to a supporting paper.
Related event: Study: Small Accuracy Gains Compound Over Long-Horizon Tasks(3 posts)→
More from AGI Musings
- Is superintelligence a bigger crypto threat than quantum? Experts split — examachine · 2026-10-16
- 'AI fakes intelligence the way a crane fakes strength' — pushback on the 'AI is a scam' take — LucaAmb · 2026-10-16
- MIT's Buehler: AI as 'a new language of science,' three years on — ProfBuehlerMIT · 2026-10-16
- Mere availability of AI advice collapses human "I don't know" from 43% to 3% — ValerioCapraro · 2026-10-16
- Uncensored AI Is About Sovereignty, Not Crime, Argues Open-Model Advocate — learnedall · 2026-10-16
- Sherry Turkle's 'Artificial Intimacy': AI is coming for our capacity for attachment — LuizaJarovsky · 2026-10-16