Karpathy on Context Windows vs Fine-Tuning
No Priors · youtube · 2026-07-14
Andrej Karpathy believes that a context window is a "cheap" way to manipulate AI behavior, but it is fundamentally different from genuinely fine-tuning or customizing a model while trying to preserve its original capabilities.
He emphasizes that the science of how to "manipulate the brain" isn't fully researched yet. Compared to simply stuffing information into the context, stably modifying a model's capabilities is far more complex.
More from AGI Musings
- Anthropic's S-1 pledges AI for curing diseases and serving 4.6B underserved patients — petergyang · 2026-10-27
- Goldman finds statistically significant negative link between AI adoption and US employment growth — ben_j_todd · 2026-10-27
- a16z: World-changing tech makes a chaotic entrance — AI agents are following the same path — a16z · 2026-10-27
- NEJM AI on medical education: mastery requires performing without the machine — BraydonDymm · 2026-10-27
- Dev warns abliterated models could wreck software supply chains with 1000+ transitive dependencies — nitarshan · 2026-10-27
- Delete your priors: why every SOTA model drop should reset how you think the world works — zack_overflow · 2026-10-27