BM25 Beats Some LLM Retrievers on MIRIAD, Sparking Debate

On the MIRIAD dataset, dense retrievers and some LLMs underperform BM25 because queries are generated directly from documents, favoring keyword matching; however, fine-tuned multi-vector models beat all dense and sparse retrievers tested.

2026-08-26 ~ 2026-08-26 · 3 related posts

• Dense retrievers lose to BM25 on MIRIAD — queries generated from docs skew the benchmark — tomaarsen · 2026-08-26
• Fine-tuned multi-vector retrieval beats dense and sparse models, benchmarks show — tomaarsen · 2026-08-26
• BM25 outperforms LLMs on MIRIAD dataset due to query generation bias — tomaarsen · 2026-08-26