Do Frontier Weights Deprecate? The Inverse-U Debate on Context vs. Internalization
agarwl_ · x · 2026-11-16
Responding to arohan's claim that "frontier weights lose value over time, and AI's economic value will increasingly come from running and improving models continuously," agarwl argues the relationship is more of an inverse-U curve: context adds value early on, but once context grows large enough, it must be internalized into weights or KV cache to keep paying off.
An accompanying chart illustrates the crossover between token-space compression (rises then falls with context scale) and parameter-space compression (keeps climbing, overtaking past the crossover point) — an exchange on where the value boundary between context and weights lies.
More from AGI Musings
- Noah Smith: Both TypeSafe AI Founders, Now Worth Billions, Spent Years 'Bumming Around' — paulnovosad · 2026-11-16
- Bindu Reddy: China lacks Blackwells, US lacks power — an AI slowdown looks inevitable — bindureddy · 2026-11-16
- The jagged frontier: fully automated coding, but still writing key docs by hand — Zergylord · 2026-11-16
- Top mathematician Tsimerman says he's quit pure math to work on AI safety — tak3sh8 · 2026-11-16
- Critics Say Effective Altruism Spends Too Little on AI Risk Messaging — AndyMasley · 2026-11-16
- Why Replacing Feynman Is Harder Than Replacing Einstein in the AI Era — CatAstro_Piyush · 2026-11-16