Retrieval Eval Trap: Strong Models Get Beaten by Weaker Ones on BEIR, DL19/20 and MIRACL

bclavie · x · 2026-11-19

Asked which retrieval benchmarks to watch, bclavie argues a genuinely strong retriever will be outperformed by a weaker-but-decent model across BEIR, DL19/DL20 and MIRACL. If a model tops all of them, it is very likely below SOTA.

The reasoning: a model better than the existing labels will "lose" on benchmarks that depend on those labels.

Related event: HF engineer: a true SOTA retriever "loses" on old benchmarks(6 posts)→

Original post →

More from Research

Research channel →