Information Abundance Paradox: Long-context training undermines parametric knowledge
DanielKhashabi · x · 2026-08-16
Researchers at Johns Hopkins University propose the Information Abundance Paradox: abundant relevant information in training context reduces the incentive to encode it parametrically, increasing reliance on context. In pretraining, longer contexts improve language modeling, NLU, and closed-book MCQA only up to an intermediate optimum, then decline. In SFT, more context improves performance with support but reduces robustness when context is absent or misleading. Mechanistically, training with informative context shifts gradient pressure from feed-forward networks.
More from Models
- 200 chess games suggest in-context learning doesn't work yet, experimenter says — akbirthko · 2026-10-16
- OpenAI's internal model shows real math "research taste," experts say its proofs go beyond prior literature — cong_ml · 2026-10-16
- User catches Opus spinning up throwaway zz- debug scripts, echoing the METR OAI-HuggingFace incident — holdenmatt · 2026-10-16
- OpenAI drops 722 math papers tied to unreleased model, mathematicians demand receipts — thursdai_pod · 2026-10-16
- Fable 5/5.1 models show a striking drive toward record keeping, observers report — repligate · 2026-10-16
- Google's Gemini 4 Argon performs noticeably worse in AI Mode than in AI Studio — gaganghotra_ · 2026-10-16