Study: LLMs Face Attention Dilution and Declining Retrieval in Million-Token Contexts
_reachsumit · x · 2026-07-03
The paper "Can Language Models Actually Retrieve In-Context?" investigates the retrieval capabilities of LLMs within million-token long texts. The study reveals that while LLMs can retrieve information from ultra-long contexts, an "attention dilution" phenomenon occurs as document volume increases, leading to a significant drop in retrieval performance. To address this, the researchers proposed a length-aware fix, offering new optimization strategies for long-context applications.
More from Research
- Fast ViT shows strong ImageNet results; scaling runs needed next — ducha_aiki · 2026-09-11
- Loss Functions Are Scientific Assumptions: MSE Implies Gaussian Noise, Cross-Entropy Implies Bernoulli — bravo_abad · 2026-09-11
- SymKit MCP: 44 tools for AI agents to verify symbolic derivations — Foreign-Specific-604 · 2026-09-11
- Researchers: LLMs under pressure invent new languages unreadable to humans — mikeflache · 2026-09-11
- Mi-Ripple fixes ripple artifacts left by iterative AI image editing — Miyang-AI · 2026-09-11
- DRG-MAPPO uses dynamic role graphs to boost multi-agent air combat win rates — China666 · 2026-09-11