Better Retrieval, Worse Robustness:How Multi-hop RAG Amplifies Upstream ASR Errors
Zhenghua Bao
EMNLP 2026 (Main Conference)
cs.CL, cs.IR, eess.AS
2026-08-24
After Whisper on four TTS accents, IRCoT+HippoRAG2's F1 gap is 36–67% larger than naive RAG on three multi-hop QA sets. Entity errors cover 87–96% of 2Wiki failures.
Voice pipelines run ASR before retrieval. Accented speech raises word error rate, and it was unclear whether extra structure in multi-hop RAG absorbs those errors or spreads them. This paper treats accent as a controlled upstream noise condition, not a fairness audit, and asks whether graph linking and iterative reformulation widen the clean-to-noisy gap.
Questions are synthesized with four Microsoft Edge TTS female voices (US, Indian, Filipino, Nigerian), transcribed by Whisper-large-v3 greedy decoding, and compared with a clean-text oracle that skips speech. Each of HotpotQA, 2WikiMultiHopQA, and MuSiQue contributes 1,000 validation questions. The generator is gpt-4o-mini.
Four retrievers, from simple to stacked:
Amplification means a larger F1 drop from oracle to high-WER accent inside one method, not a lower absolute score. A degradation case is a question the method answers at F1≥0.5 from clean text and below 0.5 from the transcript. Error types are rule-tagged with difflib; capitalized multi-word spans stand in for entities.
Two surface fixes are diagnostic probes: union retrieval over five decoding temperatures, and phonetic nearest-neighbor entity repair with Double Metaphone plus edit distance.
Every dataset-method cell is worse under accent than under oracle. Nigerian speech has the highest WER and the largest gap. Relative drops run about 15–34%. Mean WER and the F1 gap correlate at Pearson r=0.88.
| Dataset | Naive gap | Combined gap | Relative increase |
| HotpotQA | 0.104 (0.617→0.513) | 0.142 (0.730→0.588) | +36.5% |
| 2Wiki | 0.137 (0.468→0.331) | 0.195 (0.645→0.450) | +42.3% |
| MuSiQue | 0.043 (0.282→0.239) | 0.072 (0.381→0.309) | +67.4% |
Absolute F1 still favors the stacked method: 0.450 vs Naive 0.331 on Nigerian 2Wiki. Structure buys a higher ceiling and a wider clean-to-noisy drop. Either IRCoT or the graph alone widens the gap; both together widen it most. IRCoT slightly helps Naive at step 1 (0.104→0.096) and only inflates HippoRAG2 to 0.142 by step 3.
Entity corruption is the main failure mode: 87–96% of 2Wiki degradations, 67–82% on HotpotQA, 54–78% on MuSiQue. In the 2Wiki Nigerian bin with WER≥20% (413 questions, 41% of that pool), Naive degrades 25.2% of the time and the stacked method 37.8%.
N-best recovers almost nothing (-2.2% to +2.5%). Phonetic repair closes at most 11.1% of the gap, with HippoRAG2 gaining the most. 500 real Nigerian utterances show 28.9% WER, not quieter than the 17.1% synthetic 2Wiki figure. SeamlessM4T raises WER to 28.0% and preserves the method order, with the largest gap moving from 0.195 to 0.214.
A voice multi-hop stack that picks winners on clean text will pick IRCoT plus graph retrieval. Under accent, that same structure writes entity errors into follow-up queries and PageRank seeds, and later reformulation can drop leftover correct tokens. A safer deploy pattern is to enable multi-hop only at high ASR confidence and fall back to dense retrieval otherwise. Surface-form patches do not cancel structural amplification.
The main QA run is TTS with one female voice per accent, not recorded multi-hop speech. Real audio only checks ASR patterns, not end-to-end QA. The study is English-only. Entity labels are a rule-based proxy without human review. The generator is gpt-4o-mini only; a larger model inside IRCoT could behave differently. The authors stress this is not a fairness result and should not be read as a claim about speaker groups.