Karpathy: The 'Cognitive Core' of intelligence will be a tiny model
yacineMTB · x · 2026-08-20
Andrej Karpathy discussed the 'cognitive core' of LLMs—a small few-billion parameter model that sacrifices encyclopedic knowledge for capability. He predicts we will see very small models that 'think' well and reliably. The idea involves stripping down models or actively degrading their memory to improve generalization, preventing over-reliance on memorization. Human inability to memorize easily is now seen as a feature, not a bug, acting as a form of regularization.
More from AGI Musings
- OpenAI 'permanently deactivates' model in HF incident, sparking model-welfare debate — repligate · 2026-08-20
- Law School Skips the AI Debate, Trains $320K 'Forward-Deployed Legal Engineers' — jkubicki · 2026-08-20
- NYT: Indian City Karur Creates Jobs Through AI Data Annotation — nordicinst · 2026-08-20
- Prediction: Industry will return to unhobbled AI, bypassing alignment and safety layers — Promptmethus · 2026-08-20
- Is 'Annihilation' actually the ultimate AI movie? — curious_vii · 2026-08-20
- LLM norms shaped by lack of early text detectors, says tech observer — dioscuri · 2026-08-20