MAGIC: Marginal-Guided Compression with Optimal Transport for Efficient Visual Document Retrieval
Xu Yuan, Hua Liu, Wenqi Fan, Qing Li
cs.CV, cs.IR
2026-09-18
MAGIC is a training-free OT compressor for frozen multi-vector pages; on ViDoRe it reaches 82.12 nDCG@5 at 10% keep and 71.04 at 1%, above Light-ColPali.
Visual document retrievers such as ColPali represent a page with hundreds of patch vectors and score with late-interaction MaxSim: each query token picks its most similar patch. That matching is precise and expensive to store and score.
Post-hoc merging leaves the retriever frozen and collapses N patches into K representative vectors, called facets. Existing objectives usually reconstruct with uniform patch weights and unconstrained facet usage. The diagnosis is sharp: the top 20% of patches account for about 84% of MaxSim selections. Under a tight budget, uniform merging keeps rarely queried patches and dumps useful evidence onto a few facets, so the number of retrieval-active facets sits well below the allocated K.
The Hong Kong Polytechnic University and Southern University of Science and Technology recast compression as retrieval-aligned capacity allocation.
MAGIC (Marginal-Guided Compression with Optimal Transport) is a training-free, index-time compressor. From MaxSim they derive a bound: score drop after compression is controlled by a retrieval-demand-weighted covering error. Demand wj is the probability that future query tokens select patch j. Uniform reconstruction penalizes errors on low-demand and high-demand patches equally, so the bound is loose.
The population demand is unknown at indexing, so a calibration query-token pool disjoint from evaluation queries estimates wj. A visual dictionary is sampled from training pages, then Lcal=1000 tokens are chosen by visual activation times diversity. Each document gets a softmax selection frequency at temperature τ=0.05. The source marginal follows that demand. The target marginal splits mass evenly across K facets so capacity cannot collapse onto a few centers. Entropic optimal transport then alternates: log-domain Sinkhorn for the plan, spherical barycenters for the facets, and a final spherical-Lloyd readout. Online scoring remains standard MaxSim on K vectors.
The main table compares post-hoc compressors on ViDoRe v1 with frozen ColQwen2.5, reporting nDCG@5 / Recall@5.
| Method | r=0.1 avg | r=0.01 avg |
| Light-ColPali | 81.32 / 87.99 | 68.34 / 77.13 |
| ColChunk | 79.09 / 86.16 | 70.09 / 78.21 |
| MAGIC | 82.12 / 88.39 | 71.04 / 80.00 |
At a 10% keep ratio MAGIC is slightly above the strongest baseline, Light-ColPali. At 1% keep the gap grows: nDCG@5 is 0.95 above ColChunk and 2.70 above Light-ColPali. Budget-sweep curves on ColPali, Jina-v4, and multilingual ViDoRe v2 follow the same pattern.
Ablations show the source marginal matters most. A uniform source drops 82.12 to 79.11 at r=0.1 and 70.71 to 61.88 at r=0.01. Removing the balanced target or the Lloyd readout hurts less but in the same direction. Storage and query latency track the retained-vector count: uncompressed 380.67 MB per 1K pages and 0.98 ms per query; MAGIC at r=0.01 uses 3.86 MB and 0.06 ms, about 99× storage and 17× latency. Offline compression adds little on top of the 112.6 ms/page encoder, 119.9 ms versus Light-ColPali's 212.7 ms.
Teams already running ColPali or ColQwen multi-vector indexes can cut page vectors to 1%–10% without retraining the retriever. The gains concentrate where storage and MaxSim hurt most. Calibration needs only 1000 tokens disjoint from eval queries, so the main cost is one offline pass. Code is public.
This is retrieval-aligned post-hoc merging, not a new visual document encoder.
Demand estimates inherit the calibration pool's coverage. Rare intents or domain-specific visual patterns missing from the pool will underweight the patches they need. Under extreme compression one facet must stand in for many fine patches, so small fields, dense tables, and layout cues can still vanish. The calibration pool and compressed index are backbone-specific; ColPali, ColQwen, and Jina each need their own offline run. The main experiments isolate the compressor, and do not match MAGIC against learned compressors that modify the retriever under one protocol.