IRPAPERS benchmark: reranking lifts hit rate from 15% to 48%, but rerankers are now the weak stage
CShorten30 · x · 2026-11-18
The Weaviate team published IRPAPERS on arXiv, a visual document benchmark for scientific retrieval and QA. Amplifier SearchDataEng summarized results that change how search pipelines should be built and budgeted:
- The benchmark uses questions where the right document shares few words with the query — common in support, troubleshooting and research
- Keyword + embedding search alone puts the right document first 15% of the time; adding a reranker plus an LM pass raises that to 48%
- Documents are no longer lost at first-stage retrieval: with 2,000 candidates, 87% of relevant docs are in the set, but the best reranker puts only 56% of them in the top 20 — the reranker is now the weak stage
- Deeper candidate pools aren't free: one of three rerankers scored worse at k=2,000
More from Research
- Riemann hypothesis 'proven by Turkish mathematician using AI' claim mocked as the 'Organic Turk' — CatAstro_Piyush · 2026-11-22
- Andriy Burkov releases The Hundred-Page Language Models Book, hands-on with PyTorch — burkov · 2026-11-22
- Generative AI meets cardiology: Dagstuhl seminar bridges ML and clinical practice — StephanMandt · 2026-11-22
- SubRoformer hits SOTA music source separation with DSP subband filters — affige_yang · 2026-11-22
- Emily Riehl releases Lean game teaching dependent type theory at intro-to-proofs level — abhi9u · 2026-11-22
- Curated reading list: evaluating mobile-use agents on Android, from AppAgent to AndroidReality — East-Muffin-6472 · 2026-11-22