The Test of a SOTA Retriever: It Loses on Existing Benchmarks Because It Beats the Labels
KShivendu_ · x · 2026-11-19
Asked which retrieval benchmarks matter, bclavie says the true measure of a retriever pushing SOTA is that it loses on many existing benchmarks because it is better than the existing labels.
In other words, leaderboards built on old labels can no longer separate stronger models.
Related event: HF engineer: a true SOTA retriever "loses" on old benchmarks(6 posts)→
More from Research
- GPT-6.2 Astra Formalizes Never-Before-Formalized 2006 Theorem: 1,794-Line Lean Proof in 94 Minutes for $21.58 — Dr_Singularity · 2026-11-19
- n8n workflow evaluates LLMs with two cross-provider judges to avoid self-preference — ZtasnimS75 · 2026-11-19
- Meta's MLE agents ran 1,097 experiments on production rankers with 29.6% acceptance rate — dair_ai · 2026-11-19
- Survey: A Dual-Paradigm Framework for Agentic AI, Reviewing 90 Studies (2018–2025) — adnan_hashmi · 2026-11-19
- RhoAgent in Nature Biotech brings AI design from proteins to programmable functional RNA — EricTopol · 2026-11-19
- Michael Tomasello on MLST: LLMs skipped the infrastructure of shared agency — Machine Learning Street Talk · 2026-11-19