Models Appear Dumber Outside Their Strong Suits
burkov · x · 2026-07-14
The author argues that once frontier agentic LLMs step outside their typical strengths—primarily math and coding—they noticeably "get dumber".
He observed:
- In typical tasks, the model will push back on the user when necessary, take initiative, and even exhibit a degree of creativity.
- But in out-of-distribution scenarios, it often does the bare minimum superficially, lacking initiative and critical thinking.
The author concludes that if you feel the model has gotten "smarter" over the past year, it's likely just because you've been using it for in-distribution scenarios.
More from AGI Musings
- Why So Many AI Researchers Think the Machines Could Kill Everyone — connoraxiotes · 2026-09-11
- jjvincent invokes Terence Tao: ceding exploration to AI means ceding human agency — jjvincent · 2026-09-11
- OpenRouter agents now out-consume humans as AI usage arrives in three waves — AccBalanced · 2026-09-11
- If AI teleports us to solutions, how do underlying fields develop? — jjvincent · 2026-09-11
- Op-ed: the ">10% extinction" narrative is liability evasion — AI is just software, and the vendor is the defendant — gerardsans · 2026-09-11
- AI researchers just saw the power of a single resignation — and still claim there's nothing they can do — birchlse · 2026-09-11