Generative Late-Interaction Embeddings For Visual Document Retrieval
Mohamed Eltahir, Talal Aloushan, Rose Khairoalsendi, Jana Shata, Mohammed Alhassan, Leen Alrehaili, Tanveer Hussain, Naeemullah Khan
cs.IR
2026-09-11
ColPali page vectors sit on a 5-6D sphere. GLIE stores 4 vectors per page and regenerates the rest, keeping 79% of uncompressed nDCG@5 on ViDoRe v1 after a 3-minute 415K fit.
Visual document retrieval settled on late interaction. ColPali stores a vector per image patch and scores a query with MaxSim, a sum of per-query-token maximum inner products. One page is about 1,031 vectors of dimension 128, roughly 258 KB in bfloat16, so a million pages are 258 GB before the index. Accuracy comes from not collapsing a page into one vector. The bill is storage.
Published compression mostly keeps a subset or a local average of the encoder outputs. Results rarely go below about 16 vectors per page. Smaller budgets usually retrain the encoder and force a corpus re-encode. This paper asks a different question: what is the stored object, geometrically, and can the codec follow that geometry?
Across 6,729 pages and three encoders, a page's token cloud has median intrinsic dimension about 4.9 to 6.1, and every encoder L2-normalizes its outputs onto the unit sphere. Ambient dimension varies 24× from 128 to 3,072; intrinsic dimension moves by about one.
Two consequences. A k-means centroid is a Euclidean mean of unit vectors, so it sits inside the sphere and systematically understates MaxSim. Reprojecting it onto the sphere is free. And with five or six degrees of freedom, a few anchors plus one shared decoder should be able to regenerate the page.
GLIE does this post hoc with a frozen encoder:
Search is two-stage: MaxSim over the k stored vectors for every page, then decode only the top L=20 and rescore. The codec has 415K parameters.
On ViDoRe v1 (10 subsets, macro nDCG@5) the uncompressed ceiling is about 0.836. At k=4, GLIE scores 0.657, 79% of that ceiling, against about 70% for the best prior training-free method. At k=16 it is 0.759, about 91%. A million pages shrink from 258 GB to about 1.0 GB.
| Method (ViDoRe v1) | k=4 | k=16 |
| Normalized k-means (free) | 0.605 | 0.736 |
| GLIE | 0.657 | 0.759 |
| Light-ColPali LoRA, matched budget | 0.544 | 0.632 |
| Uncompressed ColPali | 0.836 | 0.836 |
Spherical anchoring alone is +0.093 nDCG@5 at k=4 and +0.030 at k=64. The learned code adds +0.044 to +0.016 up to k=16. Generative read-out adds +0.016 at k=4. Decoder size from 184K to 13M moves nDCG by at most 0.009; the smallest decoder wins at k=4. Fitting on 1,250 / 2,500 / 5,000 pages leaves the k=4 margin unchanged, so about a thousand pages saturate. The same recipe on ColQwen2 keeps 82% of uncompressed quality at k=4.
At a matched small budget, 13.3M LoRA on the encoder for about 1.5 GPU-hours never reaches free normalized clustering. GLIE, at under three GPU-minutes and 415K parameters, wins all six budgets by 0.074 to 0.132. That is not a knock on published Light-ColPali, which trains on 130K queries for about 72 GPU-hours per budget and reports mainly k≳16.
If a late-interaction stack still clusters with k-means, normalizing centroids is a one-line patch. If the goal is a handful of vectors per page without re-encoding the corpus, GLIE is a post-hoc path: freeze a public checkpoint, cache embeddings, fit per budget.
The new axis is regenerating evidence on demand, not storing fewer bits of the same vectors. The authors treat the decoder as the remaining design surface.
At k=4, perfect decoding of the same shortlist would reach 0.782 against GLIE's 0.657; that gap is decode fidelity. The remaining 0.782 to 0.836 is shortlist recall. Raising L from 20 to 100 barely moves GLIE (0.660) while the oracle climbs to 0.822: the candidates are there, the decoder does not use them. ViDoRe v1 saturates at high k; on v2 the read-out keeps paying through k=64. The gain curve is setting-dependent.
There is no composition with PLAID-style quantization. Video late interaction is named as a larger target and left untested. The paper's code link is still a placeholder.