ConceptFormer: Learning Adaptive Latent Concepts for Query-Document Alignment in Visual Document Retrieval
Peng Chunyi, Xu Zhipeng, Yan Yukun, Liu Zhenghao, Yu Shi, Mei Sen, Sun Yubo, Zhang Yongheng, Zhou Jie, Gu Yu, Yu Ge, Sun Maosong
cs.CV, cs.IR
2026-08-16
A Tsinghua-Northeastern team adds a training-time latent-concept channel to visual document retrievers: a large VLM grounds evidence regions, concept length adapts to coverage, inference carries zero overhead, and average NDCG@10 gains 16.7% relative over the strongest visual baseline.
The first step of multimodal RAG is finding query-relevant pages in a pile of document screenshots. The dominant recipe encodes the full page into one vector and trains query-page relevance with page-level contrastive learning (InfoNCE). The supervision has a blind spot: it says this page is relevant, not where. To answer "what did New Zealand score in the 2011 semifinals", a model must lock onto both the event-year context at the top and the specific number inside the central chart, and page-level supervision cannot teach that localization.
Existing fine-grained approaches split into two camps, each flawed. Text-description proxies verbalize evidence regions and lose charts and spatial structure; cropped-region proxies keep visual grounding but lack global context. The paper's position: evidence should live in a continuous middle layer, expressed in neither words nor pixels alone.
ConceptFormer bolts a training-time latent-concept channel onto a standard retriever, in four steps:
At inference the latent-concept channel shuts off entirely and the original retrieval pipeline runs, zero extra cost. That is the design's most elegant property.
Training uses about 38,000 query-document pairs; evaluation covers six benchmarks: InfoVQA in-domain, with ChartQA, SlideVQA, TQA, OWID Charts, and Wikimedia Maps out-of-domain.
| Method | Avg R@10 | Avg NDCG@10 |
| NV-Embed-v2 (strongest OCR text) | 73.66 | 62.24 |
| VisRAG-Ret (strongest visual) | 77.11 | 63.48 |
| ConceptFormer (Phi3V backbone) | 80.08 | 67.00 |
| ConceptFormer (Qwen2.5-VL backbone) | 88.33 | 75.97 |
Average NDCG@10 improves 16.7% relative over the strongest visual baseline and 22.1% over the strongest OCR baseline. The gap widens where evidence is spatially distributed: on Wikimedia Maps, NDCG@10 reaches 61.69, more than double the strongest prior visual retriever's 29.04. The weaker Phi3V backbone still beats the strongest baseline by about four points, so gains do not come purely from backbone scale.
Ablations unpack the mechanism. A textual concept proxy (74.85) and a visual concept proxy (72.81) both trail the full method (75.97), confirming latent concepts occupy a middle ground neither proxy reaches; removing the KL loss falls to 74.22, back at page-level contrastive learning. Capacity analysis shows a long-tailed length requirement: over half the samples need at most 64 tokens, median 50, while 15.1% need 129-256, 7.9% exceed 256, and P90 hits 224; every fixed-length variant underperforms adaptive allocation, and increasing length is not monotonically better, ruling out "more tokens, better". The loss weight peaks at 0.2 and declines beyond, as over-aggressive alignment erodes discriminative signal. Case visualizations show the InfoNCE-only model activating visually salient regions without connecting context, while ConceptFormer lights up both the top year banner and the central chart.
For teams building document RAG, this is a retrieval-quality upgrade with no inference-cost increase: borrow a large VLM's grounding ability during training, stay untouched at deployment. The idea of capacity that adapts to evidence coverage transfers to other representation-learning problems where required capacity varies with difficulty. Code and data are open, verified on two backbones, so the barrier to adoption is low.
The incremental positioning deserves honesty: it adds a supervision signal to existing retrievers rather than proposing a new architecture, gains range from a few points to a doubling, and concentrate in visually structured domains.
Training depends on a strong VLM for evidence proposal, whose quality caps the concepts, and sensitivity to proposal noise is untested. 38,000 pairs is small by current standards; whether adaptive-capacity gains survive at larger scale is unverified. Inference is free, but per-sample latent-concept generation happens during training, and the added training cost over plain InfoNCE is not reported. InfoVQA is in-domain while the other five are called out-of-domain, yet the training set spans five document families, so the out-of-domain spread is limited. The doubling on Wikimedia Maps partly reflects a weak baseline there (29.04), which inflates the relative gain. The method binds to patch-grid projection, and applicability to retriever architectures without regular visual grids is unaddressed.