Preprint finds longer context can weaken parametric learning in LLMs
DanielKhashabi · x · 2026-08-14
A new preprint proposes the "Information Abundance Paradox": when relevant information is abundant in the training context, the model has less incentive to internalize that information into its parameters — meaning longer context can actually weaken parametric learning in LLMs.
In other words, the more ready-made answers sit in the context, the more the model leans on copying from context rather than writing knowledge into its weights, with direct implications for long-context training and data-mixing strategies.
More from Research
- 200 chess games suggest in-context learning doesn't work yet, experimenter says — akbirthko · 2026-10-16
- Mosaic: multi-session agentic RL with hierarchical advantage estimation — LARK-Lab · 2026-10-16
- Meituan LongCat's WorldCast benchmark: GPT-Image-2.5 tops at 72.8, open-source lags — meituan-longcat · 2026-10-16
- Recursive Environment Scaling tanks six models' success rates by 57–77 points — Lingxiao Jing · 2026-10-16
- Harvard's MPT: decentralized agent swarms learn self-orchestration via bidding — Harvard · 2026-10-16
- MIT Biology and Schwarzman College hiring tenure-track professor in computational biology with generative AI focus — anshulkundaje · 2026-10-16