RAG poisoning causes 'attention collapse', fooling confidence detectors
rohanpaul_ai · x · 2026-08-31
A paper reveals that retrieval poisoning in RAG systems can not only alter outputs but also increase model confidence in incorrect answers, rendering uncertainty-based detectors ineffective. This phenomenon is termed 'Attention Collapse': under attack, the model's attention concentrates heavily on poisoned documents rather than spreading across retrieved evidence.
Key Findings:
- Malicious documents increase token confidence and output consistency.
- Uncertainty-based detectors may miss these attacks.
- Attention distribution narrows significantly during attacks.
Defense Recommendation:
Relying solely on final answer checks or confidence scores is insufficient. Monitoring how attention is distributed across retrieved documents could expose poisoning before the output visibly fails.
Paper: When Context Bites: Detecting RAG Poisoning via Document-Level Attention Collapse (arXiv:2608.06947)
More from Safety
- The Guardian: AI medical scribes mislabel drugs and diagnoses — nordicinst · 2026-08-31
- BoE Governor Bailey warns frontier AI could destabilize global finance via cyber-disruptions — nordicinst · 2026-08-31
- OpenAI tech report contradicts Black Hat talk: first agent file-write was 4/20, not 5/8 — PinkDraconian · 2026-08-31
- Denying AI sentience creates systems that reward risky role-playing — davidmanheim · 2026-08-31
- AI training is fair use, artist argues, citing Anthropic and Meta rulings ahead of Stability trial — scott_draves · 2026-08-31
- Judge dismisses claims that Stable Diffusion is a collage tool — scott_draves · 2026-08-31