sentence-transformers adds Matryoshka training for multi-vector models: 28.5 vs 18.8 nDCG@10 at 16 dims
tomaarsen · x · 2026-11-12
sentence-transformers maintainer tomaarsen highlights that a recent release quietly added Matryoshka-aware (MRL) training for multi-vector (late interaction) models via MultiVectorMatryoshkaLoss, alongside token pooling.
Key points
- Truncating token embeddings from 128 to 32 dims cuts vector storage 4x, but naive truncation hurts retrieval; MRL training computes the retrieval loss at multiple widths (e.g. 128/64/32) so leading dimensions stay useful on their own.
- His benchmarks (NanoBEIR mean nDCG@10): at 16 dims Matryoshka training scores 28.48% vs 18.76% for standard training; at 32 dims 43.44% vs 37.38%; the gap closes by 64 dims and nearly vanishes at 128 (48.95 vs 48.45).
- Practical notes: larger backbones can truncate to 64 dims without much loss, smaller ones can't; token pooling generally preserves retrieval quality better at similar storage savings, so try it first.
More from Infra
- Tesla deployed 46.7 GWh of battery storage in 2025, over 15% of the global total — XFreeze · 2026-11-12
- Baidu ships Kunlunxin M100 inference chip, M300 flagship slated for 2027 — Baidu_Inc · 2026-11-12
- Latent Sparse Attention clusters KV values for 70x speedup over full attention — bendee983 · 2026-11-12
- e-means beats k-means in 1.4 epochs, delivering 10-45x GPU speedups on embeddings — pmttyji · 2026-11-12
- Five AWS re:Invent Predictions: Bedrock Agents and Trainium Take Center Stage — DavidLinthicum · 2026-11-12
- FlagOS × MiniCPM5-2B inference throughput challenge opens with ¥30K prizes — _jaydeepkarale · 2026-11-12