Dense retrievers lose to BM25 on MIRIAD — queries generated from docs skew the benchmark
tomaarsen · x · 2026-08-26
Responding to a challenge about in-domain dense retrievers, tomaarsen shared two models scoring 67.04 and 61.42, admitting they underperform. He argues the real reason dense models can't beat BM25 on MIRIAD is that the dataset's queries were generated from the documents, giving BM25 a large lexical-overlap advantage. The linked HF model card shows his fine-tuned embeddinggemma-300m reaching 0.99 Cosine Recall@10 on retrieval.
Related event: BM25 Beats Some LLM Retrievers on MIRIAD, Sparking Debate(3 posts)→
More from Research
- Schmidt Sciences awards $10M to 19 AI humanities research teams — OdedRechavi · 2026-10-15
- Mistral and CERN partner to build a Large Physics Model trained on billions of simulated collisions — MistralAI · 2026-10-15
- Global Transport coupling lifts guided flow generation: ImageNet FID 2.06 to 1.91 — mmbronstein · 2026-10-15
- Your breathing gives you away: Wi-Fi signals can reveal your breath rhythm — TinfoilTricorn · 2026-10-15
- Beyond text correlation: attaching a geometry simulator to ground model reasoning — gerardsans · 2026-10-15
- Sequen CEO Zoe Weil: Ranking Models Should Optimize Purchases, Not Clicks — TWIML AI Podcast · 2026-10-15