RAG corpus expansion causes answer churn even when accuracy barely changes, audit method proposed
_reachsumit · x · 2026-08-25
The paper shows that expanding a RAG system's corpus can change many answers even when accuracy barely moves. They propose a Snapshot Compatibility Audit to estimate excess answer churn. In a 400-question Natural Questions study, normalized-exact and blinded-semantic excess churn are 6.44 and 10.25 percentage points, while exact-match accuracy changes by only -1.50 points.
Related event: CMU study reveals hidden answer drift in RAG systems during index updates(2 posts)→
More from Research
- OSVerify: LLM verification framework finds 18 issues across 11 OpenAI math preprints — james_y_zou · 2026-10-16
- Intelligence is proportional to data: reorder tasks by data volume and the jagged frontier becomes a spiral — sebkrier · 2026-10-16
- Doc2Harness: New benchmark finds best coding agent passes only 53% across 329 cases — _akhaliq · 2026-10-16
- Stanford's Fei-Fei Li team open-sources OpenWAM, a unified world-action model framework for robots — jiqizhixin · 2026-10-16
- NVIDIA's TacitVLA: robots reason without reading their own thoughts, 3.4x lower latency — arankomatsuzaki · 2026-10-16
- Cohere Labs world model course session 4: DreamerV1-V4 architectures in one go — Cohere_Labs · 2026-10-16