RINSE: judging RAG evidence sufficiency before generation beats relevance alone
_reachsumit · x · 2026-09-30
- Retrieved passages in RAG often name the right entities without containing the facts needed; even when instructed to abstain, 12 generators still answer 40.0–99.3% of insufficient-evidence questions.
- The authors flag pitfalls in building insufficient-evidence tests (lexical overlap and evidence position leak labels) and build a paired benchmark via substitution, deletion, and question-swap constructions that control surface features.
- RINSE is a lightweight local scorer that judges sufficiency before any generation, combining three signals: question-part coverage, whether any passage offers an answer, and a small LLM's joint reading of the passages.
- Across six datasets it ranks sufficient above insufficient evidence with a score of 0.837 (chance 0.5), catching unsupported answers without a full generator call.
More from Research
- Fleuret: teacher forcing turns time iteration into depth iteration, and GPUs love it — francoisfleuret · 2026-10-24
- Fleuret: teacher forcing leaves no room for internal state dynamics, hence clunky token reasoning — francoisfleuret · 2026-10-24
- Domingos: An attention head is just a fast-weight MLP — pmddomingos · 2026-10-24
- Ross Taylor: studying LLM behavior will increasingly resemble social science — rosstaylor90 · 2026-10-24
- Ross Taylor on RLCR: simplicity is the final achievement, details matter — rosstaylor90 · 2026-10-24
- SenseTime open-sources 27B robot model matching GPT-6-Astra at ~1/50 the cost — rohanpaul_ai · 2026-10-24