Sentence Transformers v6.3 brings token pooling-aware training for multi-vector retrieval

Sentence Transformers maintainer tomaarsen released v6.3.0. The core update brings token pooling-aware training and kernel acceleration for multi-vector (ColBERT-style late interaction) embedding models, along with a blog post and tutorial thread explaining usage.

Confirmed

Multi-vector models keep one vector per token, letting different query tokens match different parts of a document, but the number of vectors balloons; token pooling uses hierarchical clustering to merge similar document vectors into cluster means, compressing storage
tomaarsen trained two ModernBERT-base models on the same 120k MS MARCO triples, changing only the loss function design: pooling-aware training lets the compressed model retain retrieval quality—49 vectors achieve 97.8% of the retrieval quality of 193 vectors
Overall, document vectors are cut by 84.7% with only a 2.7% drop in retrieval scores
tonywu71 added that token pooling already achieves extreme compression in late-interaction multimodal retrieval such as ColPali and NeoMME, and pooling-aware training pushes it further

Why it matters

Multi-vector retrieval delivers strong quality but high storage and compute costs are the main bottleneck to adoption; this release builds compression directly into the training pipeline—loss function changes alone can slash costs—offering direct practical value for multi-vector and multimodal retrieval systems in production.

2026-11-11 ~ 2026-11-12 · 7 related posts

Primary sources

Sentence Transformers v6.3 ships token pooling-aware training — tomaarsen · 2026-11-11 23:04 (UTC+08:00)
sentence-transformers v6.3 ships token-pooling-aware training, cuts doc vectors 84.7% with 97.3% score kept — tomaarsen · 2026-11-11 23:05 (UTC+08:00)
How token pooling compresses multi-vector retrieval embeddings — tomaarsen · 2026-11-11 23:04 (UTC+08:00)

[source] Sentence Transformers v6.3 ships token pooling-aware training — tomaarsen · 2026-11-11
[source] How token pooling compresses multi-vector retrieval embeddings — tomaarsen · 2026-11-11
Pooling-aware training keeps 97.8% retrieval quality with 4x fewer vectors — tomaarsen · 2026-11-11
[source] sentence-transformers v6.3 ships token-pooling-aware training, cuts doc vectors 84.7% with 97.3% score kept — tomaarsen · 2026-11-11
Pooling-aware training cuts retrieval vectors 4x while boosting nDCG to 97.8% of unpooled — lateinteraction · 2026-11-11

2 near-duplicate retellings: CShorten30 · antoine_chaffin