ColNanoVDR: Document-Free Query Distillation for Multi-Vector Visual Document Retrieval via Optimal Transport
Zhuchenyang Liu, Ziyi Wang, Yao Zhang, Yu Xiao
cs.IR, cs.CL, cs.CV
2026-09-28
OTW distills multi-vector VDR query encoders into 149M text students that keep ~95% NDCG@5 on ViDoRe v1–v3 and run up to 26× faster on CPU.
Visual document retrieval (VDR) matches a text query against page images with no OCR. The strongest systems build both towers on 2B–9B vision-language models and score with late interaction: each query token finds its nearest page token and those maxima are summed (MaxSim). Pages are indexed offline. The query encoder still runs on every request, so a multi-billion-parameter VLM sits on the online path.
Query-side distillation swaps only that encoder and leaves the teacher's document encoder and page index untouched. The usual recipe, score distillation, trains the student to match the teacher's MaxSim on candidate pages. Every training page must be encoded first. A high-resolution page yields over a thousand token vectors. For ColVec1.1 the paper estimates about 2 TiB of cached page tokens for one million training pairs, rebuilt for every new teacher. NanoVDR already trains on teacher query embeddings alone, with no pages, but only for single-vector retrievers. On ViDoRe v3 the multi-vector teachers here lead the best single-vector system by 15 to 19 NDCG@5 points. Without a document-free recipe for multi-vector models, a compact query encoder cannot serve the strongest existing indices.
Two obstacles. Teacher and student tokenize differently, so token sets differ in size and have no correspondence. MaxSim takes a maximum over page tokens, so it is not obvious that aligning query tokens controls scores on pages never seen in training.
ColNanoVDR treats a query as a weighted set of unit-norm token embeddings. The teacher uses uniform weights 1/Kt. The student predicts a weight per token from the query context with a linear head and a softmax. On the training queries the student emits 17 tokens on average against the teacher's 29, so some student tokens have to cover several teacher tokens.
Alignment is entropic optimal transport with a cosine cost 1 − ⟨si, tj⟩. The objective is OTW (Optimal Transport with Learned Weights). The transport plan is a soft alignment matrix, like word alignment in machine translation, and needs no token-level correspondence. The loss is the soft alignment cost. The weight head gets no extra label: mass shifts toward student tokens that sit near many teacher tokens. Entropy makes the plan differentiable. A log-domain Sinkhorn solver backprops through the last iteration only. On query-sized matrices (about 17×29) training is about as fast as score distillation.
Theorem 1 shows that this alignment cost upper-bounds the MaxSim score gap on every non-empty finite page, including pages absent from training. Aligning queries is in principle enough.
At inference the teacher query encoder is dropped. Student tokens are rescaled by their weights and scored with standard MaxSim against the teacher's index. The student is an Ettin text-only encoder; the main run is Ettin-150M plus two linear heads, 149M parameters. Training uses the NanoVDR set: 711,603 pairs plus 777,649 machine-translated query variants. Teacher query tokens are cached once.
Each 149M student is scored against its own teacher's index. Retention is in parentheses.
| Teacher | v1 | v2 | v3 |
| ColQwen3.5-4.5B | 90.7 (99.0%) | 60.0 (94.2%) | 55.1 (93.8%) |
| Tomoro-ColQwen3-8B | 90.0 (99.3%) | 60.6 (93.3%) | 54.9 (93.0%) |
| Vultron-4.5B | 91.3 (99.4%) | 65.0 (96.1%) | 58.3 (95.5%) |
| ColVec1.1-4b | 90.3 (99.5%) | 64.0 (96.1%) | 59.1 (95.8%) |
| ColVec1.1-8b | 90.9 (99.3%) | 65.4 (96.5%) | 60.1 (96.0%) |
On v1 every student keeps about 99% of its teacher. On the harder v2 suite and the out-of-domain v3 enterprise collections, retention stays between 93.0% and 96.5%, at 30×–60× fewer parameters. NanoVDR-S (69M, single-vector) scores 43.5 on v3; ColModernVBert (259M) scores 17.4. The teachers lead those compact baselines by 15 to 19 points on v3; the students keep most of that lead.
On one CPU thread at batch size 1, ColNanoVDR encodes a query in 87 ms against 2290 ms for ColQwen3.5, about 26×. Every 2B–9B VLM retriever needs more than a second. On GPU the gap shrinks to 1.3–2.2×; ColQwen3.5's 110.7 ms reflects a Torch fallback for hybrid linear-attention. Ettin-32M already keeps 87% of its teacher on v3. Retention rises monotonically from 32M to 400M, with the gains on v2 and v3.
Under identical data and optimization, OTW matches the strongest document-dependent baseline, Listwise KL: on ColQwen3.5 it trails by 1.2 points on v2 (60.0 vs 61.2) and ties on v1 and v3; on Tomoro-ColQwen3-8B it leads all three, by 5.0 points on v3 (54.9 vs 49.9). InfoNCE is weaker (v3 47.1 and 46.2). OT-uniform, OTW without learned weights, stays within 1.7 points on ColQwen3.5 but drops 4.7 to 9.2 points on Tomoro. The weight head stabilizes distillation when tokenizations diverge.
OTW reads 27 GiB of cached teacher data (query tokens only). Score distillation reads 343 GiB, 12.6× more. Hierarchical token pooling on the ColVec1.1-4b v3 index at 9× compression moves retention from 95.8% to 95.3%.
If a multi-vector VDR index already exists, keep the 4B–9B page encoder and serve queries with a 149M text model on a single CPU thread. No re-indexing. A new teacher costs encoding its training queries. Training no longer caches a thousand-plus tokens per page.
Index compression and natively small retrievers do not touch this bottleneck: once the index is fixed, a large query encoder still sits online. Token pooling composes with the student.
This is query-side engineering distillation, not a new retrieval paradigm. Index size is unchanged. Search cost falls only because the query is shorter. The student is an English-centric text encoder; page understanding is borrowed from the teacher.
All students come from the Ettin family. Every configuration is a single seed; no variance is reported. On ColQwen3.5 the gaps among document-free objectives are at most two points, and the v1 gaps sit inside single-run noise. The objective comparison covers two teachers; the other three in Table 1 train OTW only, so the ranking of score distillation and fixed-weight variants under them is untested. Training queries come from a paired set even though pairing is unused; unpaired query logs are not tried. The theoretical bound is loose by a factor of about eight and does not distinguish student weightings; score distillation, which does not optimize it, retrieves comparably. ViDoRe v3 has a French subset and training includes translated queries, but results are not broken down by language.
On ColQwen3.5 the 149M student scores 55.1 NDCG@5 on v3 against the teacher's 58.7. If those 3 to 4 points matter, this student may not be enough. The 400M student reaches 56.7 on v3, closer, at 286 ms on CPU versus 87 ms.