Do Frontier Weights Deprecate? The Inverse-U Debate on Context vs. Internalization

agarwl_ · x · 2026-11-16

Responding to arohan's claim that "frontier weights lose value over time, and AI's economic value will increasingly come from running and improving models continuously," agarwl argues the relationship is more of an inverse-U curve: context adds value early on, but once context grows large enough, it must be internalized into weights or KV cache to keep paying off.

An accompanying chart illustrates the crossover between token-space compression (rises then falls with context scale) and parameter-space compression (keeps climbing, overtaking past the crossover point) — an exchange on where the value boundary between context and weights lies.

Original post →

More from AGI Musings

AGI Musings channel →