Doc-REFRAG: Rethinking Multimodal Document Retrieval-Augmented Generation
Ruofan Hu, Shengyang Xu, Minjie Hong, Xiaoda Yang, Sashuai Zhou, Ke Lei, Tao Jin, Zhou Zhao
EMNLP 2026 Main
cs.IR, cs.CV
2026-08-31
Zhejiang University compresses document images into coarse chunks and RL-expands question-relevant ones. Average accuracy is 57.5% versus 51.3% for Qwen3-VL, in 6.1s end to end.
The hard case in multimodal RAG is not a clean PDF. It is a pile of retrieved pages from mixed sources that overlap, repeat, and contradict. Document MLLMs are mostly trained on coherent pages from one file, and they drop when the context turns into retrieval noise. Visual tokens also stack per page, so twenty images blow up prefill.
Compression has been aimed at the wrong target. Attention pruning and complexity-based keepers both assume dense-looking patches matter more. In RAG, dense patches are often off-topic. This EMNLP 2026 paper from Zhejiang University changes the move: compress everything first, then restore only the chunks the question needs.
Data first. DocLongRAG starts from Doc-750K and OpenDocVQA, drops questions answerable from parameters, questions that need the web, unanswerable items, and ethically risky content. Each relevant page is then interleaved with hard negatives ranked 4 onward, with m drawn uniformly from 20 to 30. A 500-query audit found 2.47 of the top retrieved images still relevant on average, so the top 3 are discarded rather than labeled negative. The corpus is 343k QA pairs, 3.3M pages, 37.4 images per question. Relevant pages are placed before negatives; that order helps in ablation.
The model sits on DocOwl2. The encoder emits 324 tokens per page, split into non-overlapping chunks of size k. A two-layer MLP maps each chunk to one embedding. A two-layer Transformer selector, conditioned on the question, expands chosen chunks back to k tokens. Expansion is 20% at inference and sampled from 10-30% in training.
Three stages. Single-image reconstruction freezes the decoder and trains parsing plus region localization on DocStruct4M so chunk embeddings still carry text. Multi-image continual pretraining walks OpenDocVQA, then Doc-750K, then DocLongRAG, shrinking expansion from 20% to 10% and lighting up random irrelevant chunks. The selector is trained with GRPO; the reward is answer accuracy of the frozen decoder. Pseudo-labels come from ColQwen question-patch scores. All of this runs on eight RTX 3090s.
Evaluation retrieves the top 20 pages with Jina-Embeddings-V4. Recall@20 is 93.4% across six benchmarks, so recall is not the bottleneck. Default k=3, about 150 visual tokens per image.
| Method | Mean acc. | Tokens/image | TTFT |
| Qwen3-VL 8B | 51.3% | 512 | 5.91s |
| DocOwl2 7B | 48.9% | 324 | 3.94s |
| Docopilot 8B | 50.9% | 2304 | 1.81s |
| Doc-REFRAG 7B | 57.5% | 150 | 2.57s |
Per benchmark: ChartQA 61.2, SlideVQA 60.8, InfoVQA 58.7, DUDE 53.2, ViDoRe 57.3, MMDocIR 54.9. Against compressed MLLMs at a similar token budget, accuracy is more than 18 points higher.
Selector: random 38.8%, perplexity 43.7%, ColQwen 53.6% at 9.3s, RL 57.5% at 3.7s. End-to-end: top-5 into Qwen3-VL is 46.6% in 4.2s; MonoQwen2 reranking 20 to 5 is 53.2% in 15.4s; Doc-REFRAG on all 20 is 57.5% in 6.1s.
Drop reconstruction and mean accuracy falls from 55.1% to 19.7%. Shuffle the curriculum and it falls to 24.2%. Drop DocLongRAG and it falls to 48.9%. Fine-tuning InternVL3.5, MiniCPM-V-2.6, and DocOwl2 on the same corpus adds 3.9, 4.3, and 3.9 points.
The efficiency problem in multi-image document RAG is not thinner tokens. It is compression that listens to the question. On synthetic A4 pages, ANLS stays above 95% up to 4,000 characters, in the same band as DeepSeek-OCR at 256 tokens and DocOwl2 at 324, so coarsening does not have to erase text.
For deployment, 6.1s over 20 images at higher accuracy than a 15.4s rerank pipeline is a swap worth pricing. The data is independently useful: three outside models all move after fine-tuning. This is incremental systems work. The backbone is still DocOwl2; the contribution is noisy training data plus question-guided expansion.
k is frozen at training time and cannot track page density. ANLS drops about 5 points past 4,000 characters. Supporting several k means several selectors plus a router, which fights the latency goal.
The selector is rewarded only on answer correctness, with no chunk-level credit. 64% of remaining errors sit on dense or structurally complex text. Training noise is constructed; only evaluation uses a real retriever. The corpus is English. Dense CJK may want a smaller k. Results are from a single run, which is common in this subfield and still leaves no error bars.