AdaptiveEmbed: Sample-Adaptive Multi-Vector Representation for Multimodal Retrieval
Xinze Liu, Lei Yang, Dayan Wu, Hengjie Zhu, Zihao Zhang, Hanqi Wu, Tianzhu Hu, Peng Fu, Zheng Lin, Weiping Wang
cs.CV
2026-08-26
AdaptiveEmbed allocates multi-vector capacity per sample. At 2.1 tokens on average it reaches 63.45/61.14 mAP on COCO, beating the same-backbone 3-to-40 token budgets.
Multi-vector retrieval has moved from ColBERT-style MaxSim to visual sets such as ColPali and MetaEmbed. The usual debate is how to build and compare those vectors. Capacity is still a system-wide knob: one k for every sample, or one global operating point at test time.
An oracle run shows that the useful k varies across the full available range, and surface complexity does not predict it. What matters is whether one more vector helps retrieval for that sample. The paper names this setting SAMVR: capacity is a per-sample decision, and the resulting object is a content-adaptive embedding set.
AdaptiveEmbed has two stages. Stage I uses a Q-Former to emit eight candidate tokens, then trains them as two ordered groups with Multi-Group Contrastive Learning. Each group has a coarse anchor plus finer tokens. Short prefixes are compact; longer prefixes add detail. Similarity is SetSim, not MaxSim: two equal-sized sets get a maximum-weight one-to-one matching, and the mean matched similarity is the score. The same set can serve as query or gallery.
Stage II picks which tokens a sample keeps. A Token Selection Transformer sees the current set, the next legal prefix tokens, and MaxSim feedback against a retrieval bank. Each step adds one token or stops. Training enumerates the marginal AP of every legal action, standardizes it, and maximizes expected utility under the policy. At inference the process starts from the smallest prefix and stops when the expansion probability falls under δ. Only five terminal sizes are reachable: 1, 2, 4, and 8 tokens, decided in three hierarchical steps.
On four image-text datasets under full-gallery one-to-one mAP, AdaptiveEmbed uses 2.1 tokens on COCO and scores 63.45/61.14 T2I/I2T. Same-backbone MetaEmbed goes from 62.31/60.48 at 3 tokens to 62.27/59.12 at 40 tokens. Extra capacity does not help. Qwen3-VL-Embedding-8B as a single vector is 50.20/49.52. On OpenImages, with a gallery over 546K, AdaptiveEmbed leads both directions. Flickr30K I2T is the miss: 75.72 versus MetaEmbed (2/4) at 77.24. It wins six of eight image-text directions.
Video and audio keep the same recipe. ActivityNet uses 1.9 tokens and scores 61.62/56.48, above MetaEmbed, while Qwen3-VL-Embedding-8B still leads at 66.26/57.48. Clotho audio is 32.56/31.75 versus WAVE-7B at 30.65/28.47. The oracle at similar capacity is much higher: 69.51/67.85 on COCO. Dropping MGCL or bank feedback both hurt; without feedback the mean token count even rises to 2.5.
The cost of multi-vector retrieval is matching time and storage, not a guarantee that more vectors are better. Stopping near two tokens per sample keeps set matching far cheaper than a 16/64 fixed set. MetaEmbed is reproduced inside the same Stage-I training, so the gap is the allocator. Anyone paying late-interaction cost can treat per-sample k as a new knob.
It does not beat every large single-vector encoder. On video, Qwen 8B still wins. Adaptive capacity is a budget policy, not a backbone upgrade.
The learned policy still trails the oracle: 63.45 versus 69.51 T2I on COCO. Three hierarchical steps cannot express intermediate layouts, and MaxSim bank features weaken as decisions get finer.
Evaluation is one-positive full-gallery mAP, not the usual COCO five-caption Recall@K. Baselines were re-run under the same protocol, but numbers are not interchangeable with the literature. Flickr I2T loses to MetaEmbed (2/4). The policy is trained on bank utilities; a shift in the bank can shift allocation. The speed gain is an average-case effect. A hard-sample-heavy corpus would shrink it.