Information Abundance Paradox: Long-Context Training Undermines Parametric Knowledge
Arda Uzunoglu, Benjamin van Durme, Daniel Khashabi
cs.CL, cs.AI
2026-08-13
Longer training contexts make models rely on context over memorized facts: task understanding peaks near 2K tokens then declines, and turns fragile without or against context.
The dominant assumption in scaling language models is that longer training contexts are strictly better: more evidence, richer signal. This paper (Uzunoglu, van Durme, Khashabi) challenges that.
The tell comes from already-shipped models. Phi-3's 128K variant systematically loses to its 4K version on few-shot and zero-shot tasks; OLMo 3's 65K variant loses to its 8K version. The authors argue this is not just a data-quality issue, but a sign that context length itself changes what a model learns.
The central concept is the Information Abundance Paradox: when task-relevant information sits right in the training context, the model can cut loss by reading that context rather than encoding the facts into its weights. Learning shifts from parametric internalization (storing knowledge in parameters) toward contextualization (learning to use in-context evidence).
Two lines of experiments back this. For pretraining, the authors train Llama-2 models from 20M to 750M parameters on 10B tokens of Project Gutenberg, sweeping the context window from 512 to 32768, then measure language modeling, SuperGLUE, and closed-book MCQA (ARC, CommonsenseQA, PIQA). For supervised fine-tuning, they attach LoRA to Qwen3-0.6B through 14B on four MMLU-Pro domains (Health, Economics, Law, Psychology), fix a budget of n=8 documents, and vary only the number of task-relevant documents k in {0,4,8}, then evaluate under three conditions: supporting context, conflicting (misleading) context, and no context at all.
Mechanistically, the paper attacks from three angles: average training gradient norm as a proxy for solution complexity, the allocation of gradient pressure between FFN (feed-forward networks, associated with factual knowledge) and self-attention, and whether attention at inference actually lands more on context tokens.
Pretraining traces an inverted-U. SuperGLUE and closed-book MCQA peak near 2048 tokens and then decline as the window grows; language modeling peaks later, near 8192, and declines after.
| Metric | Peak context | Beyond |
| SuperGLUE / closed-book MCQA | 2048 tokens | declines |
| Language modeling | 8192 tokens | declines |
The SFT results expose the cost plainly. Raising k from 0 to 8 strengthens performance when supporting context is present, but it significantly cuts no-context accuracy and makes the model more vulnerable to misleading context. The authors call this context addiction: the richer the training context, the more the model leans on it.
The three mechanistic lines line up. On the tasks that get addicted, gradient norms fall as context lengthens, meaning context offered a simpler solution. The FFN-to-self-attention gradient ratio drops in 19 of 20 SFT comparisons and across all pretraining scales, showing optimization pressure moving from knowledge-storing FFNs to context-routing attention. At inference, these models direct more attention to context tokens, concentrated in middle layers. The clincher is module-restricted fine-tuning: updating only FFNs yields steadier no-context performance and smaller degradation under misleading context; updating only attention yields stronger supporting-context performance but worse robustness to conflict. That is causal, not just correlational, evidence for the FFN-as-memory, attention-as-context split.
This punctures the longer-context-is-always-better intuition. For practitioners, the implication is concrete: if a model ships into settings where context is absent or may be polluted, then stuffing task-relevant documents into SFT may be quietly hollowing out its parametric knowledge and amplifying its susceptibility to misdirection. It also reframes the Phi-3 and OLMo 3 regressions as possibly a property of this very mechanism rather than an engineering artifact.
Long context is a real capability. It is not, however, a neutral scaling axis: it is a switch that changes how the model learns.
The biggest limit is scale: pretraining stops at 750M parameters, and the authors concede they have not verified where the inflection lands or how severe the drop gets at larger scale, data, and compute. The comparison is not FLOP-matched either; longer windows carry a higher quadratic attention cost that is not fully separated out. Natural-language pretraining uses only one Project Gutenberg subset, SFT covers four MMLU-Pro domains, and only two architectures (Llama-2, Qwen3) are tested. Taken together, "2K is optimal" is a finding of this particular setup, not a universal rule to copy.