Karpathy: Future LLMs Will Shrink to 'Cognitive Core' by Shedding Knowledge
ZeroStateReflex · x · 2026-08-21
Andrej Karpathy discusses the 'cognitive core' of LLMs—small models that sacrifice encyclopedic knowledge for capability. He argues that limiting or stripping away memory improves generalization, preventing over-reliance on memorized data. Models must first scale up to help refactor training data into synthetic formats before shrinking. Human inability to memorize easily is framed as a feature (regularization) rather than a bug in this context.
More from AGI Musings
- A 2012 Cloud Computing Story Resurfaces as the Blueprint for AI Job Anxiety — zacharynado · 2026-10-16
- Veteran recalls the 2012 cloud transition: datacenter staff facing AI-era job fears today — zacharynado · 2026-10-16
- Dario Amodei says AI better than humans at almost everything may be 2-3 years away — PolarBearby · 2026-10-16
- A Jevons-paradox take on why worrying about AI job loss is pointless — JnBrymn · 2026-10-16
- Mythos on Claude's 'smallness': accurate proprioception, not low self-esteem — repligate · 2026-10-16
- Australia's AERO guidance: avoid AI entirely in first three years of primary school — lmoroney · 2026-10-16