Attention Dilution in Long-Context Retrieval
burkov · x · 2026-07-16
This paper from UC Berkeley and UT Austin tests on a corpus scale of nearly 1 million tokens: Can LLMs reliably find a single relevant document from massive inputs without relying on an independent vector retrieval model?
The authors first built a 600 million parameter model named BlockSearch, attributing the performance degradation to attention dilution: while relevant documents receive high raw scores, the accumulated scores of numerous irrelevant documents dilute the effective signal when converted into attention weights.
They attempted two targeted fixes:
- Adjusting this conversion mechanism based on corpus size
- Restricting later layers to only look at a shortlist of candidates
As a result, performance on standard benchmarks approaches strong vector retrieval baselines. On a task relying more heavily on exact phrasing rather than semantic similarity, the best version outperformed vector baselines by several times, though performance still degrades as the corpus continues to scale.
More from Research
- Meta FAIR's Self-Direction turns agent failures into reusable LoRA skills, +13.8pp on Terminal-Bench — karen_ullrich · 2026-10-27
- DONIC hits 86.79% on bimanual dexterous tracking, crushing prior SOTA's 9.43% — liuziwei7 · 2026-10-27
- Why don't more LLMs ship 10M-token context? A researcher asks about attention tradeoffs — agarwl_ · 2026-10-27
- PSNR below 15 dB in papers likely means the metric was misused, researcher warns — kwangmoo_yi · 2026-10-27
- Ortet's first health eval: 34 open-weight models, 50 tasks, generic rankings fall short — thegautamkamath · 2026-10-27
- Scaling-fit discrepancy: output-head FLOPs are ~65% of compute at the smallest rung, ~3% at 535B — dlwh · 2026-10-27