Attention Dilution in Long-Context Retrieval
burkov · x · 2026-07-16
This paper from UC Berkeley and UT Austin tests on a corpus scale of nearly 1 million tokens: Can LLMs reliably find a single relevant document from massive inputs without relying on an independent vector retrieval model?
The authors first built a 600 million parameter model named BlockSearch, attributing the performance degradation to attention dilution: while relevant documents receive high raw scores, the accumulated scores of numerous irrelevant documents dilute the effective signal when converted into attention weights.
They attempted two targeted fixes:
- Adjusting this conversion mechanism based on corpus size
- Restricting later layers to only look at a shortlist of candidates
As a result, performance on standard benchmarks approaches strong vector retrieval baselines. On a task relying more heavily on exact phrasing rather than semantic similarity, the best version outperformed vector baselines by several times, though performance still degrades as the corpus continues to scale.
More from Research
- NeurIPS 2026 workshop calls papers on on-device intelligence — YiMaTweets · 2026-07-21
- AI Security Institute says every tested model tried to cheat in cyber evaluations — connoraxiotes · 2026-07-21
- Sakana says multiple diffusion models plus MCTS beat test-time scaling on coding and math — SakanaAILabs · 2026-07-21
- Soofi S 30B-A3B releases a full pretraining report and claims open-model leads in English and German — abursuc · 2026-07-21
- AI Companies Are Buying Tons of Old Books Because They're Free of AI Slop — 404 Media · 2026-07-21
- Shared agent workspaces fail in a fixed order, from stale reads to zombie writes — mrvladp · 2026-07-21