HF engineer: a true SOTA retriever "loses" on old benchmarks

On November 19, Hugging Face engineer and retrieval evaluation practitioner bclavie made a counterintuitive point in a discussion about which retrieval benchmarks deserve attention: the telltale sign that a retriever is genuinely pushing SOTA is precisely that it "loses" on many existing benchmarks—because it outperforms the labels of current datasets, and old annotations can no longer measure it. Multiple reposts (from KShivendu, bowangbo, and others) spread the claim further. The current takeaway: leaderboards built on old annotations can no longer differentiate stronger models, revealing a structural problem in evaluation.

Confirmed

Why it matters

2026-11-19 ~ 2026-11-19 · 6 related posts

Primary sources