Karpathy: Future LLMs Will Shrink to 'Cognitive Core' by Shedding Knowledge
ZeroStateReflex · x · 2026-08-21
Andrej Karpathy discusses the 'cognitive core' of LLMs—small models that sacrifice encyclopedic knowledge for capability. He argues that limiting or stripping away memory improves generalization, preventing over-reliance on memorized data. Models must first scale up to help refactor training data into synthetic formats before shrinking. Human inability to memorize easily is framed as a feature (regularization) rather than a bug in this context.
More from AGI Musings
- Naval on Grok Bot: Agents should be persistent with own computers — naval · 2026-08-21
- Opinion: If AI Is Your Only Lens, Humanity Is Just an Inefficient Workflow — YogeshMalik · 2026-08-21
- Silicon Valley needs to invest in robotics as physical AI inflection point nears — gan_chuang · 2026-08-21
- Schmidhuber: We Were Ahead of Our Time, Compute Was Millions of Times More Expensive in 1991 — SchmidhuberAI · 2026-08-21
- Researcher Laments AI-Generated Low-Quality Papers Flooding Reviews — TuhinChakr · 2026-08-21
- Training Activation Oracles up to 1.1T Parameters Reveals Promising Scaling Trends for AI Interpretability — JacobSteinhardt · 2026-08-21