Models Appear Dumber Outside Their Strong Suits
burkov · x · 2026-07-14
The author argues that once frontier agentic LLMs step outside their typical strengths—primarily math and coding—they noticeably "get dumber".
He observed:
- In typical tasks, the model will push back on the user when necessary, take initiative, and even exhibit a degree of creativity.
- But in out-of-distribution scenarios, it often does the bare minimum superficially, lacking initiative and critical thinking.
The author concludes that if you feel the model has gotten "smarter" over the past year, it's likely just because you've been using it for in-distribution scenarios.
More from AGI Musings
- Understanding the AI consciousness debate: which theory of mind is behind it — msharmas · 2026-10-27
- Functionalism: if consciousness is just what the brain computes, silicon can run it too — msharmas · 2026-10-27
- iPhone case with 85,000 reviews in days: everything fake in the age of abundance — tekbog · 2026-10-27
- Whether AI can be conscious depends on your definition: five frameworks for thinking about it — msharmas · 2026-10-27
- dunkhippo33 Shares How He Thinks About Raising Kids in the AI Era — kimmonismus · 2026-10-27
- EE-trained AI educator: systems-bottleneck thinking still matters in the AI era — dunkhippo33 · 2026-10-27