Karpathy: Future LLMs Will Shrink to 'Cognitive Core' by Shedding Knowledge
ZeroStateReflex · x · 2026-08-21
Andrej Karpathy discusses the 'cognitive core' of LLMs—small models that sacrifice encyclopedic knowledge for capability. He argues that limiting or stripping away memory improves generalization, preventing over-reliance on memorized data. Models must first scale up to help refactor training data into synthetic formats before shrinking. Human inability to memorize easily is framed as a feature (regularization) rather than a bug in this context.
More from AGI Musings
- AI researcher reflects on losing her scientific identity as industry culture erodes autonomy — mariyaivasileva · 2026-10-16
- Aella draws the line on AI art: outsource labor, not creative choices — ZeroStateReflex · 2026-10-16
- AI Is Forcing Everyone Into Management—Until It Eats Middle Management Too — MonkZer0 · 2026-10-16
- Shogun 2 remaster review-bombed over AI-upscaled graphics, sparking dev debate — draginol · 2026-10-16
- Pedro Domingos: Everyone must have their own unique model, anything else is dangerous — pmddomingos · 2026-10-16
- Interp researcher pushes back: if you can't simulate a neuron, you haven't explained it — nabla_theta · 2026-10-16