Practical guide to multi-vector embeddings: MaxSim retrieval and shrinking the index
tomaarsen · x · 2026-08-18
Tom Aarsen and two LightOn authors published a companion blog covering multi-vector (late-interaction) embedding models in practice:
- How it works: unlike dense models that compress text into one vector, multi-vector models keep a vector per token and score with MaxSim — alignment is non-lexical (live matches inhabit at 0.94).
- Gains vs costs: LightOn's LateOn (multi-vector) vs DenseOn on the same 149M ModernBERT backbone wins 9 of 13 NanoBEIR datasets (0.6868 vs 0.6764 mean nDCG@10), but the index is 16x larger (311.5 MB for 4,874 NQ passages).
- Shrinking: HierarchicalTokenPooling clusters token vectors with Ward linkage; poolfactor=2 halves the index at 100.6% of unpooled BEIR performance. Alternatives: a real late-interaction index or reranker-only use.
- Beyond text: ColQwen-Omni takes text, images, audio & video — retrieving recorded conversations zero-shot with no transcription; ColPali-style VLMs match text queries against page images, no OCR.
- Unified API: v6.0 loads PyLate, Stanford-NLP ColBERT and ColPali-style checkpoints through one class, with PyLate capabilities designed into ST alongside its authors.
Related event: Sentence Transformers v6.0 ships with first-class late interaction models(33 posts)→
More from coding & agent
- PHAROS: An Open-Source npm for MCP Servers, Written in Go — Nofear001 · 2026-08-19
- Using AI Agents to draft release reports from evidence collections — CodeByPoonam · 2026-08-19
- MUON: An Open-Source Shared Brain for Parallel Coding Agents — Virtual_Gift_5327 · 2026-08-19
- Netlify integrates OpenRouter to enable model swapping without code changes — thisiskp_ · 2026-08-19
- Dev runs three Codex accounts plus Claude to parallelize coding agents — ChanceKelch · 2026-08-19
- DeepSeek open sources 'deepseek-harness' agent framework with 130k+ stars — alex_verem · 2026-08-19