UCSD paper: context rot is attention competition, not length — rival statements slash accuracy to 11-21%
dair_ai · x · 2026-11-04
A UC San Diego paper shows long-context retrieval failures aren't about length. Adding 32K neutral filler tokens left Qwen2.5-1.5B at 97-100% accuracy, but a few same-format statements with rival values dropped accuracy to 0.11-0.21, multiplying loss 9-110x. Across 126,473 trials, attention mass left on the evidence explained up to 58% of loss variance vs at most 8% for length. A masked-filler protocol confirms causality: masking fillers hurt MMLU but in retrieval masking cost 46 points while visible filler cost none. Context rot is attention competition. Code and data released.
More from Models
- LLM routers audited: none of 7 beats a static mid-priced model at 85.3% for $0.0043 — dl_weekly · 2026-11-04
- Google's Gemini denies ties to Google, claims to be an OpenAI assistant — conitzer · 2026-11-04
- MOSS AMA highlights: image models, local speech, audio/video research and open-source culture — Physical_Hat4022 · 2026-11-04
- 128GB unified RAM laptop benchmarks: gemma4:e4b hits 122.8 TPS locally via Ollama — ollama · 2026-11-04
- Artificial Analysis new evals: Gemini 3.8 Flash ranks worst on slop and sycophancy — Immediate_Simple_217 · 2026-11-04
- OpenAI Agents API brings Computer Use to your agents with 2-3 lines of config — gabrielchua · 2026-11-04