Karpathy on Context Windows vs Fine-Tuning
No Priors · youtube · 2026-07-14
Andrej Karpathy believes that a context window is a "cheap" way to manipulate AI behavior, but it is fundamentally different from genuinely fine-tuning or customizing a model while trying to preserve its original capabilities.
He emphasizes that the science of how to "manipulate the brain" isn't fully researched yet. Compared to simply stuffing information into the context, stably modifying a model's capabilities is far more complex.
More from AGI Musings
- Gary Marcus says LLMs still cannot really do math on their own — GaryMarcus · 2026-07-22
- Gary Marcus says LLM math skills are like knowing only a car’s engine size — GaryMarcus · 2026-07-22
- AI may make digital work infinitely leveraged while offline life gets more human — illscience · 2026-07-22
- Better AI math could save researchers time by killing false conjectures earlier — prateekj · 2026-07-22
- AI’s economic forecasts are split by nearly a quadrillion dollars by 2035 — bittingthembits · 2026-07-22
- Open source is becoming tech’s soft power, says Kevin Xu — kevinsxu · 2026-07-22