Information Abundance Paradox: Long-Context Training Can Hurt Short-Task Performance

A Johns Hopkins University team's new preprint names the "Information Abundance Paradox": when relevant information is sufficiently abundant in the training context, a model's incentive to internalize knowledge into its parameters drops, and it leans on ready-made context instead. In pretraining, enlarging the context window improves language modeling, natural language understanding, and closed-book multiple choice only up to an intermediate optimum before declining; training on long contexts beyond a threshold even hurts short-context task performance — a direct challenge to the "longer context is always better" intuition.

Confirmed

• The paradox comes from Johns Hopkins researchers; NLP researcher Daniel Khashabi posted original tweets and reposts introducing the preprint across Aug 14–16, and burnytech also shared it.
• Core mechanism, per Khashabi: during long-context training, relevant knowledge easily appears in the context, so models tend to use it directly, suppressing parametric internalization of knowledge and increasing dependence on context.
• Pretraining findings: enlarging the context window follows a rise-then-fall curve on language modeling, natural language understanding, and closed-book multiple choice, with sustained decline past the optimum.
• Phenomenon: training models on long contexts beyond a certain threshold degrades their performance on short-context tasks.

Unconfirmed

• The posts indicate the preprint also examines the supervised fine-tuning (SFT) stage, but the repost text is truncated, so the specific SFT conclusions are not fully visible in the material.
• Concrete threshold values, model and data scale, and other experimental details are not disclosed in the posts.

Why it matters

• It directly challenges the industry habit of ever-lengthening training contexts: if long contexts erode parametric knowledge, closed-book knowledge and short-task performance may trade off against context length.
• It informs pretraining data construction: training may need to balance context length against knowledge internalization, controlling the abundance of relevant information in the corpus.

2026-08-14 ~ 2026-08-16 · 5 related posts

• Information Abundance Paradox: Long-Context Training Undermines Parametric Knowledge — burny_tech · 2026-08-14
• Preprint finds longer context can weaken parametric learning in LLMs — DanielKhashabi · 2026-08-14
• [source] Information Abundance Paradox: long-context training can hurt short-task performance — DanielKhashabi · 2026-08-15
• [source] Study Finds Long-Context Training Harms Short-Task Performance — DanielKhashabi · 2026-08-15
• Information Abundance Paradox: Long-context training undermines parametric knowledge — DanielKhashabi · 2026-08-16