EigenLI: Spectral Approximations to Late Interaction
Archish S, Sabyasachi Basu, Ankit Garg, Ravishankar Krishnaswamy, Kirankumar Shiragur
cs.IR, cs.LG
2026-09-07
EigenLI compresses each ColBERT document into a top-k eigenspace; at k=32 it lifts BEIR nDCG@10 by 15% over k-means++ on ColBERTv2 and yields an 8256-d vector that beats MUVERA.
Late-interaction retrievers such as ColBERT keep a vector per token and score a query with MaxSim: each query token takes its best inner product against the document tokens, then those maxima are summed. The quality gap versus single-vector dense retrieval is real. So is the cost. ColBERTv2 stores 512 vectors of 128 dimensions per document; ColQwen3 uses about 1250 vectors of 320 dimensions for a page screenshot. Index size and MaxSim latency dominate.
The strongest training-free compressors cluster token vectors into fewer centroids. k-means++ and Ward hierarchical pooling lead that comparison. Clustering still answers the wrong question. It shrinks a set of vectors into a smaller set, and ignores the fact that those vectors already occupy a thin subspace.
EigenLI starts from a spectral observation. Token embeddings of one document concentrate in a low-dimensional subspace: the second-moment matrix \(MD=\sumi di di^T\) has rapidly decaying eigenvalues. Compression then means keeping the top-k eigenvectors \(w1,\ldots,wk\) of that matrix, not picking representative tokens.
Scoring changes with the representation. Instead of MaxSim, EigenLI measures how much of each query token lands in the document's principal subspace:
\[s=\sumi\sumj\langle qi,wj\rangle^2=\sumi\|\Pi qi\|^2\]
If a query token points along directions the document actually occupies, the score is high. When most of the document mass already sits in those k directions, this score moves with MaxSim. Dropping the eigenvalues is a deliberate choice. Weighting by eigenvalues collapses the score into a sum of squared similarities over every query-document token pair, which underperforms MaxSim.
The same construction is exactly a single-vector dot product. Map each vector through the quadratic polynomial kernel \(K(x)\) and the dimension becomes \(d(d+1)/2\). For ColBERTv2, \(d=128\), so EigenLI-SV is 8256-dimensional, independent of k, and can sit in an ANN index.
On 13 BEIR development sets at k=32, the three text models do not agree.
| Model | Baseline | nDCG@10 relative lift (arithmetic mean) |
| ColBERTv2 | k-means++ / Ward | +15.0% / +8.0% |
| AnswerAI-ColBERT-small | k-means++ / Ward | +11.9% / +2.6% |
| GTE-ModernColBERT | k-means++ / Ward | −1.9% / −10.2% |
Against full MaxSim, EigenLI-32 loses about 3.2% nDCG@10 on ColBERTv2, 13.3% on AnswerAI, and 14.8% on GTE. Raising k is not free: at k=64, half of ColBERTv2's 128-d space, the subspaces become less discriminative.
On ViDoRe-v3 with ColQwen3-4B, EigenLI-32 beats k-means++ / Ward by 5.5% / 3.4% nDCG@10 (arithmetic mean over 8 datasets) and trails full MaxSim by 5.5%.
The single-vector gap is larger. EigenLI-SV at 8256 dimensions versus MUVERA's 10240-d FDE lifts ColBERTv2 nDCG@10 by 69.6% geometric / 78.9% arithmetic mean of per-dataset ratios. On AnswerAI and GTE, even after centering MUVERA, the arithmetic-mean lifts are 178.1% and 156.0%. The authors' MUVERA numbers on MS MARCO sit below the original paper, so those relative percentages are inflated. Absolute scores are the safer read.
Offline compression is cheaper too. Across 39 model-dataset pairs, k-means++ takes 6.81× longer than EigenLI on average, Ward 17.46×. EigenLI-SV also accepts FAISS PQ. At 2 bits, mean nDCG loss versus full MaxSim on four small BEIR sets is 4.4% / 4.7% / 4.2% at cutoff 10 / 100 / 1000.
This is a drop-in, training-free compressor for systems that already run a ColBERT-family model. Compute a per-document eigenspace offline, keep 32 directions instead of 512 token vectors, and both storage and scoring get cheaper. EigenLI-SV is the variant that plugs into existing ANN stacks without a MaxSim engine.
It is not a blanket replacement for Ward. On GTE-ModernColBERT, tokens sit in a narrow cone, and spectral compression at k=32 loses to clustering. The relevant axis is embedding geometry, not the compression ratio.
Almost all numbers come from brute-force scans. There is no EigenLI-SV plus ANN candidate generation plus full-MaxSim rerank bake-off against PLAID or a Ward pipeline. Latency, index size, and memory at corpus scale are still open.
EigenLI-SV grows as \(d^2\). 8256 dimensions is tolerable for ColBERT; ColQwen3's 320-d tokens would produce 51360 dimensions, which the paper did not run. Quantization covers only four small BEIR sets. On GTE, raising k from 8 to 32 can hurt datasets such as DBPedia-Entity, a sign that a degenerate low-rank geometry makes the extra directions noise. The large MUVERA relative gains partly reflect a weak reproduction, and should not be read as a production delta.