HF engineer: a true SOTA retriever loses on many benchmarks—old labels can't measure it
bclavie · x · 2026-11-19
Hugging Face engineer bclavie offers a counterintuitive take: the real sign your retriever is pushing SOTA is that it loses on many existing benchmarks, because it's better than the existing labels—stale annotation quality turns leaderboard scores into inverted signals.
The observation underscores that retrieval evaluation datasets urgently need refreshing.
Related event: HF engineer: a true SOTA retriever "loses" on old benchmarks(6 posts)→
More from Research
- GPT-6.2 Astra Formalizes Never-Before-Formalized 2006 Theorem: 1,794-Line Lean Proof in 94 Minutes for $21.58 — Dr_Singularity · 2026-11-19
- n8n workflow evaluates LLMs with two cross-provider judges to avoid self-preference — ZtasnimS75 · 2026-11-19
- Meta's MLE agents ran 1,097 experiments on production rankers with 29.6% acceptance rate — dair_ai · 2026-11-19
- Survey: A Dual-Paradigm Framework for Agentic AI, Reviewing 90 Studies (2018–2025) — adnan_hashmi · 2026-11-19
- RhoAgent in Nature Biotech brings AI design from proteins to programmable functional RNA — EricTopol · 2026-11-19
- Michael Tomasello on MLST: LLMs skipped the infrastructure of shared agency — Machine Learning Street Talk · 2026-11-19