AdaMerge: Tuning-Free Patch Compression for Multi-Vector Visual Document Retrieval
Jianxin You, Kun Ni
cs.CV, cs.IR
2026-09-19
Patch-clustering merge cosines show a cliff whose location concentrates across 11,273 documents; cutting per document removes PtM's tuning and beats it on long-document retrieval.
Multi-vector visual document retrieval (VDR) models such as ColPali encode every page into hundreds to thousands of patch embeddings and score queries with MaxSim over all of them. Accuracy is strong; the bill is not. A single page occupies tens of KB, and a modest corpus runs to hundreds of MB uncompressed. Compression either prunes unimportant patches or merges similar ones. The strongest recent method, Prune-then-Merge (PtM), reliably beats pruning-only baselines at high compression, but it needs a cluster budget m tuned by grid search on every dataset. Two authors from Montreal found a way to delete that tuning step entirely.
The observation comes first. Run hierarchical clustering over a page's patches and track the cosine similarity between the centroids merged at each step. This merge-cosine sequence decreases monotonically and shows a sharp cliff, with mergeable redundancy to the left and retrieval signal to the right. Across 11,273 documents from 14 datasets, the cliff lands in a narrow band (per-dataset medians between 0.77 and 0.85). The merge boundary is a property of each document, not a dataset-level hyperparameter.
AdaMerge:
Two global hyperparameters (cmin and the pruning factor k) are shared across every dataset and both backbones.
Long-document benchmark ViDoRe-V2, ColQwen2.5 backbone:
| Compression | AdaMerge | Tuned PtM | Uncompressed |
| 75% | 0.593 | 0.552 | 0.595 |
Dropping three quarters of the vectors costs 0.002 nDCG@5 against no compression, while grid-tuned PtM falls to 0.552. On ColNomic-3B the lead over PtM reaches +0.036 in the 85 to 90 percent range. Wilcoxon paired tests over 1,344 queries on both backbones put AdaMerge ahead at every operating point, by +0.012 to +0.038, all p < 10^-4.
On the short-document benchmark ViDoRe-V1 (10 datasets), every merging method is already near-lossless, and AdaMerge matches tuned PtM with no tuning at all. The ablation attributes +0.039 to the adaptive cut and a further +0.009 to attention weighting. Runtime is about 10 ms per document on a single CPU thread, 1 to 2 ms slower than PtM.
Deployment cost is what keeps multi-vector retrieval out of production, and this removes the last hidden cost of the strongest compressor: re-tuning per dataset and per backbone. Teams already running ColPali-family models for document RAG get a plug-and-play index compressor, and the gains grow with document length and redundancy, exactly where storage hurts most.
Gains concentrate in long documents; on short ones there is no accuracy advantage over PtM. cmin is global but still hand-set, and the paper offers no theory for why the cliff exists. Evaluation covers only ColPali-family encoders. The 1 to 2 ms per-document overhead compounds when building very large indexes.