Retrieval eval insight: models topping BEIR/DL19/DL20/MIRACL are likely below SOTA
bclavie · x · 2026-11-19
Embeddings developer bclavie shares a fun MIRACL example: a GPT model in a listwise setting doesn't even beat Qwen3 Rerank 8B (or 4B) on the release labels.
He offers a broader rule of thumb:
- A very good model will often be outperformed across all of BEIR, DL19/DL20, and MIRACL by a weaker-but-decent model
- Conversely, if a model tops all those benchmarks, it's very likely below SOTA in practice
Takeaway: single-benchmark leaderboards are a poor proxy for real retrieval quality, and he teases building a cross-domain retrieval evaluation benchmark to fix this.
Related event: HF engineer: a true SOTA retriever "loses" on old benchmarks(6 posts)→
More from Models
- FoodTruck Bench, an LLM agent business-sim benchmark, lands on Papers with Code — NielsRogge · 2026-11-19
- GPT's Ultrafast mode is 'really good' but burns through tokens, says dev — lxfater · 2026-11-19
- Hermes adapter solves a SWE-Bench Pro task in 35 tool calls where the base Qwen3.8 model failed — yb2698 · 2026-11-19
- NousResearch's Hermes adapter lifts Qwen3.8-27B agent scores: SWE-Bench Pro 29.4%→43.1% at 4× fewer tokens — yb2698 · 2026-11-19
- Gemini 4 Neon hailed as 'most tasteful model ever' for its essay on thriving in the AI age — daniel_mac8 · 2026-11-19
- GPT-6.2 Astra's Backlash vs Its Efficiency Numbers: What OpenAI's Enterprise Push Signals — The AI Daily Brief · 2026-11-19