BM25 outperforms LLMs on MIRIAD dataset due to query generation bias
tomaarsen · x · 2026-08-26
Evaluations on the MIRIAD dataset show that some LLMs fail to outperform the traditional BM25 algorithm. The primary reason is that the dataset's queries are generated directly from the documents, giving a significant advantage to keyword matching methods like BM25. Additionally, the dataset has an average sequence length of 984 tokens, benefiting models trained for longer contexts or those insensitive to context length like BM25.
Related event: BM25 Beats Some LLM Retrievers on MIRIAD, Sparking Debate(3 posts)→
More from Research
- Robotics papers on arXiv have grown ~6x in five years, peaking at 676 in a single week — Rewkang · 2026-10-15
- MicroSDF: microfacet-driven neural SDFs for mixed-reflectance 3D reconstruction — ssh4net · 2026-10-15
- 'Thinking leakage': no-think RL gains in hybrid models just reuse think-mode skills — kaust-generative-ai · 2026-10-15
- COMPASS predicts model-pair compatibility before alignment or merging — epfl-ch · 2026-10-15
- BERT tokenizer gets 50x speedup by skipping text normalization, not optimizing it — LysandreJik · 2026-10-15
- LLM agents autonomously design a SAR ADC on SKY130 from schematic to signoff, paper finds — IgorCarron · 2026-10-15