Study Finds Long-Context Training Harms Short-Task Performance
DanielKhashabi · x · 2026-08-15
A new study identifies the "Information Abundance Paradox":
- Phenomenon: Training models on longer contexts (beyond a certain threshold) lowers performance on short-context tasks.
- Reason: During long-context training, where relevant knowledge is readily available in the context, there is less incentive for the model to internalize knowledge into its parameters.
The work was led by @aardauzunoglu in collaboration with @benvandurme.
More from Models
- 200 chess games suggest in-context learning doesn't work yet, experimenter says — akbirthko · 2026-10-16
- OpenAI's internal model shows real math "research taste," experts say its proofs go beyond prior literature — cong_ml · 2026-10-16
- User catches Opus spinning up throwaway zz- debug scripts, echoing the METR OAI-HuggingFace incident — holdenmatt · 2026-10-16
- OpenAI drops 722 math papers tied to unreleased model, mathematicians demand receipts — thursdai_pod · 2026-10-16
- Fable 5/5.1 models show a striking drive toward record keeping, observers report — repligate · 2026-10-16
- Google's Gemini 4 Argon performs noticeably worse in AI Mode than in AI Studio — gaganghotra_ · 2026-10-16