Retrieval Community Clarifies Late Interaction Myths: Index Size Is No Longer the Bottleneck

Several retrieval developers pushed back on two common misconceptions about the Late Interaction paradigm: misused terminology and an overstated index-size bottleneck. Their conclusion: the correct name is Late Interaction with the scoring function at its core; index size has not been a problem since 2021, and the real bottleneck is infrastructure adoption rather than technical feasibility.

Confirmed

• @lateinteraction and @tomaarsen both argued that the community's widespread use of "multi-vector retrieval" is a significant misnomer; the paradigm should be called "Late Interaction," and its essence lies in the scoring function, not the number of vectors. @tomaarsen added that the name MultiVectorEncoder (MVE) is even more misleading than Late Interaction.
• @antoinechaffin ran the numbers on MS MARCO (8.8M documents): with ColBERT at 32 tokens per document, 128 projection dimensions, and binarized document-side storage, the index is about 4.5 GB—nearly half the cost of int8 dense-vector storage, inverting the intuition.
• @antoinechaffin clarified that "32 tokens" refers to the post-pooling size, not the document itself; BEIR benchmarks show hpool@5 effectiveness at 32 pooled tokens, and experiments show no performance drop even down to 5 tokens.
• @tomaarsen noted that in more realistic 1k-token document settings, earlier efficiency calculations for token pooling reverse: in long-document scenarios token pooling becomes highly attractive and effectively offsets compute costs.
• @lateinteraction pointed out that since 2021 index size has not been a problem outside visual search cases like ColPali; ColBERTv2 and PLAID solved storage costs before text-davinci-002 existed. For many applications, a standard PLAID index is actually smaller than the oversized single-vector representations in common use.

Why it matters

• Misused terminology risks obscuring the essence of the technique and diverting attention from the scoring function, its true core.
• Measured numbers and token pooling experiments show that storage and long-document cost arguments no longer hold: with the right tooling, ColBERT is viable in most scenarios where simple indexes suffice.
• The real bottleneck is diffusion, not technology: many still rely on raw matrices and MaxSim computation, ignoring more efficient implementations such as KV cache and token pooling.

2026-08-25 ~ 2026-08-25 · 9 related posts

• [source] Misnaming 'Late Interaction' as 'Multi-vector' obscures the real issue — lateinteraction · 2026-08-25
• Tech discussion: Why it should be 'Late Interaction' not 'Multi-vector' — tomaarsen · 2026-08-25
• [source] Dev on retrieval: Index size hasn't been an issue since 2021 — lateinteraction · 2026-08-25
• Vector Retrieval Myth: Index Size is Not the Bottleneck — lateinteraction · 2026-08-25
• Index size not the bottleneck; underutilization of infra is — lateinteraction · 2026-08-25
• [source] Doing the math: int8 dense embeddings cost ~2x the storage of binary ColBERT — antoine_chaffin · 2026-08-25
• Algorithm efficiency shifts with doc length; token pooling gains edge in 1k token setting — tomaarsen · 2026-08-25
• Token Pooling Can Offset RAG Costs for Long Documents — antoine_chaffin · 2026-08-25
• ColBERT cost-effectiveness debate in retrieval — antoine_chaffin · 2026-08-25