Fine-tuned ColBERT Beats General Retrievers on Medical Search
Tom Aarsen, author of sentence-transformers, published a blog post on August 26 titled "Training and Finetuning Multi-Vector Embedding Models with Sentence Transformers," along with the open-source finetuned model mLateOn-medical. Using medical retrieval as a hands-on case study, he demonstrated the complete workflow for finetuning a ColBERT-style multi-vector model on a single RTX 3090. The post also covers the MultiVectorEncoder, the fourth model type introduced in v6.0, which natively supports late interaction retrieval training.
Confirmed
- Evaluation setup: 47 model configurations were evaluated on 1,000 held-out medical questions against a corpus of 200,000 passages, spanning dense, sparse, lexical, and multi-vector architectures.
- The finetuned Late Interaction model reached NDCG@10 of 91.4, beating the best zero-shot model across all architectures by 6.2 points; the strongest dense model, Qwen3-Emb, was also included in the comparison (and did not surpass the finetuned model).
- With identical training and backbone, Late Interaction outperformed its Dense counterpart by 12 points.
- Training recipe: pretrained checkpoint, 1 million domain pairs, in-batch negative sampling, full document length, and a higher-than-usual learning rate; total training took 14.5 hours on a single RTX 3090.
- Data efficiency: training with only 100k pairs (roughly 1.2%10% of the full set) for 75 minutes yielded NDCG@10 just 1.2 points below the full 1-million-pair run, with most of the gain achieved in the first hour.
- The finetuned model achieved a top-1 hit rate of 84.9%, a clear improvement over zero-shot.
Not Yet Confirmed
- Retrieval researcher srchvrs raised a critique: some models in Aarsen's comparison are out-of-domain, so the conclusion that "dense retrieval can't beat BM25" may not hold; a dense retriever trained on the same dataset is needed as a control. This objection awaits a response or follow-up experiments.
Why It Matters
- It shows that a single consumer-grade GPU (RTX 3090) plus a modest amount of domain data can significantly outperform large general-purpose zero-shot models, dramatically lowering the barrier to building vertical-domain retrieval systems.
- The MultiVectorEncoder in sentence-transformers v6.0 brings ColBERT-style multi-vector retrieval training and finetuning into the mainstream toolchain, with direct practical value for RAG and enterprise search deployments.
- The debate itself is a useful reminder: when comparing retrievers, distinguish in-domain from out-of-domain models to avoid misleading conclusions.
2026-08-26 ~ 2026-08-26 · 13 related posts
Primary sources
- [source] Fine-tuning ColBERT for Medical Retrieval Beats General-Purpose Models — tomaarsen · 2026-08-26
- Eval: New Model Hits 91.4 NDCG@10, Leading by 6.2 Points — tomaarsen · 2026-08-26
- [source] Medical retrieval: Fine-tuned Late Interaction hits 91.4 NDCG@10 — tomaarsen · 2026-08-26
- Training Recipe: Strongest Zero-Shot Model Beats General Retrievers — tomaarsen · 2026-08-26
- Late Interaction outperforms dense models by 12 points in tests — tomaarsen · 2026-08-26
- Scaling experiment: 10k pairs, 75 mins training, 1.2 points short of full run — tomaarsen · 2026-08-26
- Medical retrieval model trained on RTX 3090 rivals full dataset runs — tomaarsen · 2026-08-26
- 14.5 hours on one RTX 3090: finetuned multi-vector retriever beats all general-purpose models — tomaarsen · 2026-08-26
- [source] Retrieval expert pushes back: out-of-domain dense models losing to BM25 proves nothing — srchvrs · 2026-08-26
- PyLate Merged into Sentence Transformers for Better Retrieval — lateinteraction · 2026-08-26
- Fine-tune ColBERT on one RTX 3090 to beat general-purpose retrievers — lateinteraction · 2026-08-26