Retrieval eval veteran: models topping BEIR/DL/MIRACL are likely below SOTA
bclavie · x · 2026-11-19
Practitioner @bclavie discusses limitations of retrieval benchmarks:
- Scarcity of document retrieval benchmarks: ViDoReV3 is the gold standard but saturates around 69% due to annotation limits; models scoring above 65% are often meaningfully stronger than they look.
- Classic benchmark paradox: a very good model will typically be outperformed on all of BEIR, DL19/DL20 and MIRACL by a weaker-but-decent model. If a model tops all of those, it is very likely below SOTA.
- At lower param counts things are iffier — methods like LateInteraction/mLateOn/SparseUP are very strong in their categories.
Related event: HF engineer: a true SOTA retriever "loses" on old benchmarks(6 posts)→
More from Models
- Doctor says worrying about LLM citation hallucination means you're 2 years behind — flngr · 2026-11-19
- Mystery stealth model appears on benchmark leaderboard, beating Fable 5.2 — gajus0 · 2026-11-19
- Gemini Neon reportedly matches Opus quality with 3-4x the quota of Claude or Codex — s_scardapane · 2026-11-19
- GPT-6.2 Astra Formalizes Never-Before-Formalized 2006 Theorem: 1,794-Line Lean Proof in 94 Minutes for $21.58 — Dr_Singularity · 2026-11-19
- Frontier ranking debate: OpenAI credibly third for the first time as Anthropic's lead narrows — teortaxesTex · 2026-11-19
- Microsoft Claims Its Router Is '79x Cheaper' Than GPT-Sol 6.1—By Its Own Test — YvesMulkers · 2026-11-19