Retrieval expert pushes back: out-of-domain dense models losing to BM25 proves nothing
srchvrs · x · 2026-08-26
Retrieval researcher srchvrs challenged: where is a dense retriever trained on the same dataset? The models used were seemingly out-of-domain, so failing to beat BM25 proves little. Context: he had just published a blog fine-tuning a ColBERT-style multi-vector model for medical retrieval with Sentence Transformers, beating every general-purpose retriever he could find after 14.5 hours on a single RTX 3090. The exchange with tomaarsen's reply highlights a methodology issue: how a benchmark's queries are constructed (from documents or not) can systematically favor BM25.
Related event: Fine-tuning ColBERT on a single RTX 3090 beats general-purpose retrievers(14 posts)→
More from Research
- Intelligence is proportional to data: reorder tasks by data volume and the jagged frontier becomes a spiral — sebkrier · 2026-10-16
- Doc2Harness: New benchmark finds best coding agent passes only 53% across 329 cases — _akhaliq · 2026-10-16
- Stanford's Fei-Fei Li team open-sources OpenWAM, a unified world-action model framework for robots — jiqizhixin · 2026-10-16
- NVIDIA's TacitVLA: robots reason without reading their own thoughts, 3.4x lower latency — arankomatsuzaki · 2026-10-16
- Cohere Labs world model course session 4: DreamerV1-V4 architectures in one go — Cohere_Labs · 2026-10-16
- GPT-6.1 Sol audit finds 6% of arXiv papers had false main theorems, down to 1.5% recently — nitarshan · 2026-10-16