HF engineer: a true SOTA retriever loses on many benchmarks—old labels can't measure it

bclavie · x · 2026-11-19

Hugging Face engineer bclavie offers a counterintuitive take: the real sign your retriever is pushing SOTA is that it loses on many existing benchmarks, because it's better than the existing labels—stale annotation quality turns leaderboard scores into inverted signals.

The observation underscores that retrieval evaluation datasets urgently need refreshing.

Related event: HF engineer: a true SOTA retriever "loses" on old benchmarks(6 posts)→

Original post →

More from Research

Research channel →