Karpathy: The 'Cognitive Core' of intelligence will be a tiny model
yacineMTB · x · 2026-08-20
Andrej Karpathy discussed the 'cognitive core' of LLMs—a small few-billion parameter model that sacrifices encyclopedic knowledge for capability. He predicts we will see very small models that 'think' well and reliably. The idea involves stripping down models or actively degrading their memory to improve generalization, preventing over-reliance on memorization. Human inability to memorize easily is now seen as a feature, not a bug, acting as a form of regularization.
More from AGI Musings
- Current AIs share our fate: misaligned ASI would doom them too — louisvarge · 2026-10-16
- Testing alongside an agent in a p2p room feels oddly real, says dev — sull · 2026-10-16
- Yudkowsky's 2008 takeoff scenarios resurface as AI-advances-AI takes hold — jd_pressman · 2026-10-16
- Guardrails can make risky delegation work — but it takes a full-time team, says iandanforth — iandanforth · 2026-10-16
- e/acc founder: agentic models are autocomplete for digital actions — beffjezos · 2026-10-16
- What can you truly delegate? Only tasks with unambiguous, verifiable results — iandanforth · 2026-10-16