Information Abundance Paradox: Long-Context Training Undermines Parametric Knowledge
burny_tech · x · 2026-08-14
A new paper introduces the 'Information Abundance Paradox', revealing that during long-context pre-training, abundant relevant information in the context can reduce the model's incentive to encode that information parametrically, shifting its reliance toward contextualization.
The study shows that increasing the context window improves performance only up to an intermediate optimum, after which it consistently declines. During fine-tuning, richer train-time context improves results with supporting context but reduces robustness when context is absent or misleading at test time. Mechanistically, informative context shifts gradient pressure from feed-forward networks (linked to parametric knowledge) toward attention modules, as it provides a lower complexity solution.
More from Research
- 200 chess games suggest in-context learning doesn't work yet, experimenter says — akbirthko · 2026-10-16
- Mosaic: multi-session agentic RL with hierarchical advantage estimation — LARK-Lab · 2026-10-16
- Meituan LongCat's WorldCast benchmark: GPT-Image-2.5 tops at 72.8, open-source lags — meituan-longcat · 2026-10-16
- Recursive Environment Scaling tanks six models' success rates by 57–77 points — Lingxiao Jing · 2026-10-16
- Harvard's MPT: decentralized agent swarms learn self-orchestration via bidding — Harvard · 2026-10-16
- MIT Biology and Schwarzman College hiring tenure-track professor in computational biology with generative AI focus — anshulkundaje · 2026-10-16