HF engineer: a true SOTA retriever "loses" on old benchmarks
On November 19, Hugging Face engineer and retrieval evaluation practitioner bclavie made a counterintuitive point in a discussion about which retrieval benchmarks deserve attention: the telltale sign that a retriever is genuinely pushing SOTA is precisely that it "loses" on many existing benchmarks—because it outperforms the labels of current datasets, and old annotations can no longer measure it. Multiple reposts (from KShivendu, bowangbo, and others) spread the claim further. The current takeaway: leaderboards built on old annotations can no longer differentiate stronger models, revealing a structural problem in evaluation.
Confirmed
- bclavie stated explicitly that a genuinely strong retrieval model will often be beaten across an entire suite like BEIR, DL19/DL20, and MIRACL by a "weaker but decent" model; if a model dominates these leaderboards, it is likely not SOTA.
- He noted that document retrieval benchmarks are scarce, with ViDoReV3 being the current gold standard but with an annotation pipeline ceiling of around 69%; models scoring above 65% are often more capable than their scores suggest, as scores are capped by annotation quality.
- Reposters (such as KShivendu and bowangbo) summarized the implication: leaderboard ceilings are locked in by historical annotation quality, and old labels have become a drag on stronger models.
Why it matters
- This view exposes a structural problem in retrieval evaluation: benchmark scores can invert relative to real capability—the model topping the charts may not be stronger, while the one "losing" on old leaderboards may be the true SOTA.
- It has direct implications for model selection and paper review: judging retrievers solely by BEIR/DL/MIRACL rankings can be misleading, and scores should be interpreted against the ceiling imposed by annotation quality.
2026-11-19 ~ 2026-11-19 · 6 related posts
Primary sources
- [source] HF engineer: a true SOTA retriever loses on many benchmarks—old labels can't measure it — bclavie · 2026-11-19
- A truly SOTA retriever loses on existing benchmarks because it outperforms the labels — bo_wangbo · 2026-11-19
- The Test of a SOTA Retriever: It Loses on Existing Benchmarks Because It Beats the Labels — KShivendu_ · 2026-11-19
- [source] Retrieval Eval Trap: Strong Models Get Beaten by Weaker Ones on BEIR, DL19/20 and MIRACL — bclavie · 2026-11-19
- [source] Retrieval eval veteran: models topping BEIR/DL/MIRACL are likely below SOTA — bclavie · 2026-11-19
- Retrieval eval insight: models topping BEIR/DL19/DL20/MIRACL are likely below SOTA — bclavie · 2026-11-19