Fine-tuning ColBERT on a single RTX 3090 beats general-purpose retrievers

On August 26, Tom Aarsen, author of sentence-transformers, published a blog on training and fine-tuning multi-vector embedding models and open-sourced the resulting mLateOn-medical model. As a hands-on case, he fine-tuned a ColBERT-style model for medical retrieval on a single RTX 3090; sentence-transformers v6.0 adds a fourth model type, MultiVectorEncoder, with native late-interaction support via PyLate integration.

Confirmed

Evaluation: 47 model configurations (dense, sparse, lexical, multi-vector) tested on 1,000 held-out medical questions over a 200k-passage corpus.
The fine-tuned Late Interaction model reached 91.4 NDCG@10, beating the best zero-shot model across all architectures by 6.2 points; the strongest dense model, Qwen3-Emb, was included but did not surpass it.
Under identical training and backbone, Late Interaction outscored its Dense counterpart by 12 points.
Recipe: pre-supervised checkpoint, 1M domain pairs, in-batch negatives, full document length, higher-than-usual learning rate; 14.5 hours on one RTX 3090.
Data efficiency: 75 minutes on 100k pairs came within 1.2 NDCG@10 points of the full 1M-pair run, with most gains in the first hour.
Fine-tuned top-1 hit rate reached 84.9%, a clear jump over zero-shot.

Unconfirmed

Retrieval researcher srchvrs argued some baselines were out-of-domain, so claims like "dense retrieval loses to BM25" may not hold without a dense retriever trained on the same dataset; no response or follow-up experiment yet.

Why it matters

A consumer-grade GPU plus modest domain data can clearly beat large general-purpose zero-shot models, lowering the barrier for domain-specific retrieval; an hour of training captures most of the gain.
v6.0's MultiVectorEncoder brings ColBERT-style retrieval into a mainstream toolkit, directly useful for RAG and enterprise search.
The debate is a reminder to distinguish in-domain from out-of-domain models when comparing retrievers.

2026-08-26 ~ 2026-08-27 · 14 related posts

Primary sources

Fine-tuning ColBERT for Medical Retrieval Beats General-Purpose Models — tomaarsen · 2026-08-26 21:55 (UTC+08:00)
14.5 hours on one RTX 3090: finetuned multi-vector retriever beats all general-purpose models — tomaarsen · 2026-08-26 21:55 (UTC+08:00)
Retrieval expert pushes back: out-of-domain dense models losing to BM25 proves nothing — srchvrs · 2026-08-26 22:03 (UTC+08:00)

[source] Fine-tuning ColBERT for Medical Retrieval Beats General-Purpose Models — tomaarsen · 2026-08-26
Eval: New Model Hits 91.4 NDCG@10, Leading by 6.2 Points — tomaarsen · 2026-08-26
Medical retrieval: Fine-tuned Late Interaction hits 91.4 NDCG@10 — tomaarsen · 2026-08-26
Training Recipe: Strongest Zero-Shot Model Beats General Retrievers — tomaarsen · 2026-08-26
Late Interaction outperforms dense models by 12 points in tests — tomaarsen · 2026-08-26
Scaling experiment: 10k pairs, 75 mins training, 1.2 points short of full run — tomaarsen · 2026-08-26
Medical retrieval model trained on RTX 3090 rivals full dataset runs — tomaarsen · 2026-08-26
[source] 14.5 hours on one RTX 3090: finetuned multi-vector retriever beats all general-purpose models — tomaarsen · 2026-08-26
[source] Retrieval expert pushes back: out-of-domain dense models losing to BM25 proves nothing — srchvrs · 2026-08-26
PyLate Merged into Sentence Transformers for Better Retrieval — lateinteraction · 2026-08-26
Fine-tune ColBERT on one RTX 3090 to beat general-purpose retrievers — lateinteraction · 2026-08-26

3 near-duplicate retellings: tomaarsen · aaxsh18 · lateinteraction