CMU paper shows retrieval skew fools RAG into picking frequent near-miss answers; JCA fixes it
pratyushmaini · x · 2026-10-26
CMU researchers Jennifer Hsia, Zachary Lipton and Aarti Singh present an EMNLP 2026 paper on a subtle RAG failure: a competing answer that satisfies only some of the question's requirements but appears frequently in retrieved documents can look best-supported, causing the model to pick it even when the correct answer is in the evidence.
Key points:
- Directly manipulating retrieval skew shows that increasing passages favoring the competing answer makes models more likely to select it
- This is a decision problem candidate-level support alone cannot resolve; candidates must be compared against the question's full requirements
- The proposed Joint Constraint Aggregation (JCA) compares candidates against shared question-derived requirements using the retrieved evidence
- Across three evaluation settings and three instruction-tuned backbones, JCA improves accuracy, with gains growing as retrieval favors the competing answer
More from Research
- Debate: could exponential compute replace wet-lab experiments in biology? — cjmaddison · 2026-10-26
- LLMs are obsessed with Japan: 24-language study traces cultural bias to SFT — alex_verem · 2026-10-26
- Top-b sampling: entropy-adaptive decoding beats Top-p on GPQA and cuts variance 10x — cneuralnetwork · 2026-10-26
- MathDB launches as an open platform for AI-era math research with Lean support — lpachter · 2026-10-26
- ConformWeb: LLM-generated web apps pass only 11.7% of behavioral contract checks — algo_diver · 2026-10-26
- Claude runs 53 hours unsupervised, beats human designs on 7 of 8 quantum problems — FinanceYF5 · 2026-10-26