VLM page annotations lift long-document retrieval F1 from 0.254 to 0.546 across five retrievers

What the Reranker Sees: Multi-Aspect Page Annotation for Long-Document Multimodal Question Answering

Guanchen Wu, Jiayuan Ding, Subhabrata Mukherjee, Carl Yang

cs.AI

2026-08-15

Precomputed VLM page annotations double long-document retrieval F1 (BGE-M3 0.254 to 0.546, ColPali 0.332 to 0.581) across five candidate pools.

What problem this solves

Long-document VQA over hundred-plus-page mixed documents usually follows retrieve-then-read. This paper shows the bottleneck has moved: on MMLongBench-Doc, BGE-M3 reaches Recall@20 of 0.86, so the evidence pages are usually in the top 20, yet F1@5 is only 0.254, and even the visual retriever ColPali manages 0.332. The hard part is picking the few true evidence pages out of 20 superficially similar candidates. The obvious fix, an LLM reranker between pool and selection, barely works when it sees only raw text snippets: MRR on BGE-M3 stays flat (0.524 to 0.523) and drops on ColPali (0.692 to 0.598), because text-only judgment overrides the visual ranking signal. The evidence that matters is locked in tables, charts, and layout, which snippets do not carry.

Method

Trident keeps the cheap text-LLM reranker and makes visual content legible to it offline.

Annotation layer (Trident-R-Index): one offline pass per document. Concepts on three axes: topic (clustering sliding-window page embeddings), entity (an extraction LLM pulls entities and numbers, co-occurrence-clustered into bundles), structure (section-tree leaves). Per page, four fields: section path, a VLM-written visual caption of up to 60 words covering tables, charts, and layout, the top-15 entity tags, and a 600-character text snippet. Annotations are cached and reused across queries.
Rerank layer (Trident-R): each candidate becomes a structured record (retriever score, concept hits, the four fields), serialized as JSON; a single LLM call adaptively selects up to 5 pages.
Generation layer (Trident-S): the VLM answers the same retrieved pages under three parallel lenses (topic, entity and value, structure); a fourth synthesis call merges them, preferring the entity view for numeric questions and the structural view for location questions. If all three lenses agree the question is unanswerable, the system says so.

The reranker is GPT-4.1, captions come from GPT-4o-mini, and nothing is trained.

Results

Configuration (MMLongBench-Doc)	F1@5	MRR
BGE-M3 raw top-5	0.254	0.524
BGE-M3 + snippet-only LLM rerank	0.374	0.523
BGE-M3 + caption-only + rerank	0.518	0.692
BGE-M3 + full annotation + rerank	0.546	0.744
ColPali raw top-5	0.332	0.692
ColPali + full annotation + rerank	0.581	0.788

The same annotation and rerank protocol lifts all five candidate pools (BM25, BGE-M3, RRF, ColPali, a multi-axis graph pool) above the strongest adaptive-K baseline, PageIndex at 0.480. The ColPali gain is the informative one: ColPali already encodes page images at retrieval time, yet annotation still lifts F1 from 0.332 to 0.581. Captions are not merely filling in missing visuals for a text retriever; they are an evidence representation the reranker can read. End to end, ColPali with annotation, rerank, and Trident-S scores highest, with rankings consistent across two LLM judges (kappa 0.913). LongDocURL transfers the trend, with Trident-R top on retrieval F1 at 0.398.

Why it matters

Generating VLM page descriptions for text pipelines is a common industry pattern that academia has largely dismissed as lossy. This paper shows that with structured, multi-field annotation, a training-free text-LLM reranker matches or beats expensive multimodal rerankers, at $0.256 per 100 pages of one-time indexing amortized over all queries. For long-document RAG this is a directly copyable architecture. Trident-S is conditional: it helps only open-ended synthesis questions (+1.7 to +6.6 accuracy) and is bypassed by design on MCQ and short-extractive workloads.

Limitations

The authors note the offline indexing pass costs money, that the five-pool isolation runs only on MMLongBench-Doc, and that LongDocURL is an external transfer check. Two more from reading it: most of the lift comes from the visual caption alone, with the topic, entity, and structure axes adding about 0.03 jointly, so the complexity-to-gain ratio is poor and the authors themselves suggest a caption-first variant when other fields are expensive; and the reranker and captioner are GPT-4.1 and GPT-4o-mini, with smaller or open models claimed as drop-in substitutes but not measured.

Terms

Retrieve-then-read: a retriever selects pages, then a model answers from them.
Recall@20 / F1@5: gold-page coverage of the top 20; F1 of the selected 5 against gold.
ColPali: a retriever that encodes rendered page images with a vision-language model.
MRR: mean reciprocal rank of the first hit.
Adaptive K: the model decides how many pages to select instead of a fixed top-5.