CoverageBench-Negate: Dense Retrievers Fail Near Chance When Evidence Contradicts Their Priors
_reachsumit · x · 2026-10-29
Coverage Under Contradiction: A Benchmark for Negation-Aware Retrieval (arXiv:2610.37162)
- Question: can retrievers still find corpus evidence when it contradicts their parametric knowledge? Existing coverage benchmarks use claims retrievers may already know.
- Method: for each nugget in CoverageBench, create a negated counterpart and edit its supporting documents to support the negation while preserving context.
- Scale: CoverageBench-Negate spans 6 datasets and 63,000 original-edited evidence pairs, with construction metadata, out-of-family LLM judgments, and a human audit.
- Baseline result: BM25 separates near-twin edited evidence far better than dense retrievers (Qwen3, Jina), which stay near chance.
- Released on Hugging Face Datasets along with the prompts used to build it.
More from Research
- Cross-lingual LLM effects paper wins Outstanding Paper award at EMNLP 2026 — anas_ant · 2026-10-29
- Lie detector compiled into Qwen3.5 weights flags 91% of lies with zero false positives — teortaxesTex · 2026-10-29
- Practitioner: No serious lab actually trains on the test set today — nrehiew_ · 2026-10-29
- Researchers slam secrecy around OpenAI-linked Yang-Mills proof materials — anshulkundaje · 2026-10-29
- Job-market paper: German engineering societies lifted regional patents 4.8% — Afinetheorem · 2026-10-29
- François Fleuret Skeptical of Self-Improvement Training: Only Boosts the Easy Early Part — francoisfleuret · 2026-10-29