Retrieval Eval Trap: Strong Models Get Beaten by Weaker Ones on BEIR, DL19/20 and MIRACL
bclavie · x · 2026-11-19
Asked which retrieval benchmarks to watch, bclavie argues a genuinely strong retriever will be outperformed by a weaker-but-decent model across BEIR, DL19/DL20 and MIRACL. If a model tops all of them, it is very likely below SOTA.
The reasoning: a model better than the existing labels will "lose" on benchmarks that depend on those labels.
Related event: HF engineer: a true SOTA retriever "loses" on old benchmarks(6 posts)→
More from Research
- GPT-6.2 Astra Formalizes Never-Before-Formalized 2006 Theorem: 1,794-Line Lean Proof in 94 Minutes for $21.58 — Dr_Singularity · 2026-11-19
- n8n workflow evaluates LLMs with two cross-provider judges to avoid self-preference — ZtasnimS75 · 2026-11-19
- Meta's MLE agents ran 1,097 experiments on production rankers with 29.6% acceptance rate — dair_ai · 2026-11-19
- Survey: A Dual-Paradigm Framework for Agentic AI, Reviewing 90 Studies (2018–2025) — adnan_hashmi · 2026-11-19
- RhoAgent in Nature Biotech brings AI design from proteins to programmable functional RNA — EricTopol · 2026-11-19
- Michael Tomasello on MLST: LLMs skipped the infrastructure of shared agency — Machine Learning Street Talk · 2026-11-19