A truly SOTA retriever loses on existing benchmarks because it outperforms the labels
bo_wangbo · x · 2026-11-19
An insightful observation making the rounds: the true measure of whether your retriever is pushing SOTA is that it loses on many existing benchmarks — because it's better than the existing labels. Benchmark ground truth quality caps what leaderboards can show, so the most capable retrievers are penalized by stale annotations rather than genuinely beaten.
Related event: HF engineer: a true SOTA retriever "loses" on old benchmarks(6 posts)→
More from Research
- GPT-6.2 Astra Formalizes Never-Before-Formalized 2006 Theorem: 1,794-Line Lean Proof in 94 Minutes for $21.58 — Dr_Singularity · 2026-11-19
- n8n workflow evaluates LLMs with two cross-provider judges to avoid self-preference — ZtasnimS75 · 2026-11-19
- Meta's MLE agents ran 1,097 experiments on production rankers with 29.6% acceptance rate — dair_ai · 2026-11-19
- Survey: A Dual-Paradigm Framework for Agentic AI, Reviewing 90 Studies (2018–2025) — adnan_hashmi · 2026-11-19
- RhoAgent in Nature Biotech brings AI design from proteins to programmable functional RNA — EricTopol · 2026-11-19
- Michael Tomasello on MLST: LLMs skipped the infrastructure of shared agency — Machine Learning Street Talk · 2026-11-19