Karpathy on Context Windows vs Fine-Tuning
No Priors · youtube · 2026-07-14
Andrej Karpathy believes that a context window is a "cheap" way to manipulate AI behavior, but it is fundamentally different from genuinely fine-tuning or customizing a model while trying to preserve its original capabilities.
He emphasizes that the science of how to "manipulate the brain" isn't fully researched yet. Compared to simply stuffing information into the context, stably modifying a model's capabilities is far more complex.
More from AGI Musings
- New paper finds AI agents negotiate badly under uncertainty and break deals on purpose — xuanalogue · 2026-10-26
- NBER: Only 2.9% of US consumers pay for AI; retention as low as 20%, pricing power at risk — avicgoldfarb · 2026-10-26
- AI researchers guessed robots wouldn't outrun humans until 2034 — it already happened — ben_j_todd · 2026-10-26
- ESR on the Bugpocalypse: locking down frontier models only hands the game to Chinese open weights — colinmcnamara · 2026-10-26
- AI folks, stop saying "biology" when you mean drug development — KevinKaichuang · 2026-10-26
- Anthropic: junior tech hires fell from 27% to 9% as agent hours hit 6x employee hours — thione · 2026-10-26