Attention Dilution in Long-Context Retrieval
burkov · x · 2026-07-16
This paper from UC Berkeley and UT Austin tests on a corpus scale of nearly 1 million tokens: Can LLMs reliably find a single relevant document from massive inputs without relying on an independent vector retrieval model?
The authors first built a 600 million parameter model named BlockSearch, attributing the performance degradation to attention dilution: while relevant documents receive high raw scores, the accumulated scores of numerous irrelevant documents dilute the effective signal when converted into attention weights.
They attempted two targeted fixes:
- Adjusting this conversion mechanism based on corpus size
- Restricting later layers to only look at a shortlist of candidates
As a result, performance on standard benchmarks approaches strong vector retrieval baselines. On a task relying more heavily on exact phrasing rather than semantic similarity, the best version outperformed vector baselines by several times, though performance still degrades as the corpus continues to scale.
More from Research
- PNAS paper shows a tiny billiard-ball system is a universal computer — undecidability lives in two dimensions — eigensteve · 2026-09-11
- New paper: Absolute pose estimation from affine cues and gravity direction — ducha_aiki · 2026-09-11
- LoMa Paper Ships REALLY HardPairs Dataset, Accepted at ECCV 2026 — ducha_aiki · 2026-09-11
- Johns Hopkins Launches Full-Stack Hands-on Robot Learning Class with SO-101 Arm Kits — _krishna_murthy · 2026-09-11
- SyncWorld: In-Context Robot World Model Simulates Unseen Views and Embodiments Zero-Shot — ChongZzZhang · 2026-09-11
- A 3D Pose Dataset for Dogs Released — ducha_aiki · 2026-09-11