Retrieval eval insight: models topping BEIR/DL19/DL20/MIRACL are likely below SOTA

bclavie · x · 2026-11-19

Embeddings developer bclavie shares a fun MIRACL example: a GPT model in a listwise setting doesn't even beat Qwen3 Rerank 8B (or 4B) on the release labels.

He offers a broader rule of thumb:

Takeaway: single-benchmark leaderboards are a poor proxy for real retrieval quality, and he teases building a cross-domain retrieval evaluation benchmark to fix this.

Related event: HF engineer: a true SOTA retriever "loses" on old benchmarks(6 posts)→

Original post →

More from Models

Models channel →