RAG corpus expansion causes answer churn even when accuracy barely changes, audit method proposed
_reachsumit · x · 2026-08-25
The paper shows that expanding a RAG system's corpus can change many answers even when accuracy barely moves. They propose a Snapshot Compatibility Audit to estimate excess answer churn. In a 400-question Natural Questions study, normalized-exact and blinded-semantic excess churn are 6.44 and 10.25 percentage points, while exact-match accuracy changes by only -1.50 points.
Related event: CMU study reveals hidden answer drift in RAG systems during index updates(2 posts)→
More from Research
- Paper: Are Pathology Foundation Model Latent Representations Invariant to Rotation? — iScienceLuvr · 2026-08-25
- Researcher calls for 2026 update to key paper, eyeing pathology domain — iScienceLuvr · 2026-08-25
- Author shares personal workflow for benchmarking models — 0xsachi · 2026-08-25
- EgoStandard: Egocentric Video Dataset Trends on Hugging Face — LightwheelAI · 2026-08-25
- Microsoft Open Sources Thinkingbox: Stateful Agent Workflow Benchmark — microsoft · 2026-08-25
- AAAI 2027 review debate: Should empirical papers without code be auto-rejected? — SimpleObvious4048 · 2026-08-25