The Test of a SOTA Retriever: It Loses on Existing Benchmarks Because It Beats the Labels

KShivendu_ · x · 2026-11-19

Asked which retrieval benchmarks matter, bclavie says the true measure of a retriever pushing SOTA is that it loses on many existing benchmarks because it is better than the existing labels.

In other words, leaderboards built on old labels can no longer separate stronger models.

Related event: HF engineer: a true SOTA retriever "loses" on old benchmarks(6 posts)→

Original post →

More from Research

Research channel →