Attention Dilution in Long-Context Retrieval
burkov · x · 2026-07-16
This paper from UC Berkeley and UT Austin tests on a corpus scale of nearly 1 million tokens: Can LLMs reliably find a single relevant document from massive inputs without relying on an independent vector retrieval model?
The authors first built a 600 million parameter model named BlockSearch, attributing the performance degradation to attention dilution: while relevant documents receive high raw scores, the accumulated scores of numerous irrelevant documents dilute the effective signal when converted into attention weights.
They attempted two targeted fixes:
- Adjusting this conversion mechanism based on corpus size
- Restricting later layers to only look at a shortlist of candidates
As a result, performance on standard benchmarks approaches strong vector retrieval baselines. On a task relying more heavily on exact phrasing rather than semantic similarity, the best version outperformed vector baselines by several times, though performance still degrades as the corpus continues to scale.
More from Research
- Researchers identify "Forking": train loss falls while validation loss turns up — ZimingLiu11 · 2026-10-26
- Google DeepMind pilots world's first double-blind evaluations for frontier models — scychan_brains · 2026-10-26
- Rewriting scaling law study: optimal synthetic data share rises from 10% to 32% with compute — heghbalz · 2026-10-26
- FLASH: GPU-native deformable sim learns cloth folding in minutes on one RTX 5090 — ChongZzZhang · 2026-10-26
- LEMON: zero-shot LLM misbehavior monitor flags hallucinations from attention maps — yoavartzi · 2026-10-26
- Behind the EMNLP keynote: LMs as simulated learners to generate child language acquisition hypotheses — najoungkim · 2026-10-26