Same Agent, Different Answers: A Repeat-Aware Audit of Corpus-Induced Answer Churn in Retrieval-Augmented QA
Jingjie Ning, Xueqi Li
cs.IR, cs.CL
2026-08-24
Holding the model fixed while FineWeb grows, CMU subtracts repeat noise from answer changes. NQ semantic excess churn is 10.25 pp while EM moves -1.50 pp; 40/400 are stable flips.
A retrieval-augmented QA system can change its answers after an index expansion even when the requested model, prompt, retrieval policy, evidence depth, and generation controls stay fixed. Aggregate accuracy hides those swaps when gains and losses cancel. Ordinary decoding noise makes one-shot before/after comparisons overstate the update. This CMU paper names the hidden effect accuracy-blind answer churn and audits it after subtracting same-snapshot repeat disagreement.
The Snapshot Compatibility Audit draws two independent answers per question per corpus snapshot. Excess answer churn D̂ is within-snapshot repeat agreement minus cross-snapshot agreement. D̂>0 means answers move more across snapshots than they already vary under repeats. Two kernels: normalized exact match, and a blinded semantic-equivalence judge that never sees scale, evidence, or correctness. The confirmatory set is a preregistered 400-question Natural Questions cohort. A frozen FineWeb prefix grows from one shard to seven; every other interface control is locked. A separately preregistered 200-question TriviaQA study is supportive. Evidence is locked before generation, so repeats measure generator noise. The semantic judge labels all 28 pairs among eight anonymous answers before gold aliases are unlocked. The NQ gate requires exact D̂ at least 3 percentage points and positive one-sided 95% lower bounds for both kernels.
On NQ, normalized-exact D̂ is 6.44 percentage points (lower bound 4.56) and semantic D̂ is 10.25 (7.69), while exact-match accuracy moves only −1.50 points. TriviaQA is smaller and the same sign: 3.00 and 2.125 points, with EM +1.25. A post-hoc strict stable flip (within-state semantic agreement at both ends, all four cross pairs disagree) hits 40/400 NQ questions and accounts for 10.00 of the 10.25 semantic gap. Among 800 NQ matched repeats, 46 go from EM-match to nonmatch and 34 the other way. The net is −1.50 points, yet gross correctness flow is 10%, and 19.38% move between two different EM-nonmatches that accuracy never sees. Mean top-eight document overlap at the endpoints is 1.19 of 8. An outcome-blind 100-question V4-Pro subset still shows 8.75 pp semantic excess churn while EM rises 3.00 points. A cross-family judge agrees on 95.71% of 1,400 audit pairs.
| Study | Semantic excess churn | Endpoint EM change |
| NQ 400 (V4-Flash) | 10.25 pp | −1.50 pp |
| TriviaQA 200 | 2.125 pp | +1.25 pp |
| NQ 100 (V4-Pro) | 8.75 pp | +3.00 pp |
An index rebuild can be a behavioral release even when weights and the public API do not change. Caches, regression tests, and downstream workflows will see different answers while the EM dashboard stays green. Retrieval-augmented releases should report excess answer churn next to utility. The protocol is cheap: two independent calls per question per state, plus a blind semantic comparison. The paper stops at the audit. It does not offer a universal ship/no-ship threshold.
The treatment is one frozen nested FineWeb path, so shard order and content are confounded; this is not a law that more corpus implies more churn. Evidence is one generator family, one search service, and English open-domain QA. Closed-book EM is higher at every scale, so the work does not show a utility gain from expansion. The semantic judge has no human gold; a 50-question cross-family audit is not a substitute. Two repeats per state are the minimum noise baseline and a coarse view of each conditional output distribution. Single-turn retriever-generator results do not transfer to multi-step agent trajectories.