A truly SOTA retriever loses on existing benchmarks because it outperforms the labels

bo_wangbo · x · 2026-11-19

An insightful observation making the rounds: the true measure of whether your retriever is pushing SOTA is that it loses on many existing benchmarks — because it's better than the existing labels. Benchmark ground truth quality caps what leaderboards can show, so the most capable retrievers are penalized by stale annotations rather than genuinely beaten.

Related event: HF engineer: a true SOTA retriever "loses" on old benchmarks(6 posts)→

Original post →

More from Research

Research channel →