Does More Retrieved Evidence Help Visual Retrieval-Augmented Generation with Diffusion Language Models?
Jiankun Wang, Yisen Gao, Ziwei Zhang, Xingcheng Fu, Jiaxin Bai, Chen Gao
cs.CL, cs.CV
2026-08-07
More retrieved pages hurt diffusion-LM visual RAG, as parallel denoising splices conflicting sources. Training-free ECF restores 2.62 points on average across three DLMs and five VQA benchmarks.
Visual retrieval-augmented generation (RAG) rests on a default assumption: more retrieved evidence raises the chance of covering the answer page, so all retrieved pages should be passed to the generator. For autoregressive models this roughly holds. This paper finds that for diffusion language models (DLMs) the assumption inverts: retrieving more pages does raise answer-page recall, yet accuracy often drops.
A DLM does not generate left to right like GPT; it denoises every unresolved position in the answer in parallel. When several retrieved pages condition those positions jointly, a position-wise proposal can splice together visual information from different, mutually incompatible sources, assembling a "hybrid" answer that no single page actually supports. The authors prove (Theorem 1) that under a factorized proposal the mass of such hybrid answers, Q∥(ℋ|E)=1−(1−ϵ)^r−ϵ^r, grows monotonically with the conflict width r. More conflicting pages, more amplified error.
Crucially, this damage is visible early. On ChartQA, adding a conflicting page drops first-step answer-block accuracy from 65.0 to 41.1 and produces the largest Jensen-Shannon divergence from the correct-source distribution.
ECF (Entropy-Based Candidate Filter) is a training-free evidence-admission framework. Its core idea is to use the DLM's own first-step block entropy to decide which candidates enter the context.
Each probe needs only a single first-step forward pass, never full denoising, and depends on no retriever scores or learned parameters, hence training-free.
Three multimodal DLMs (LLaDA2.0-Uni, LLaDA-V, Dream-VL) and five visual QA benchmarks (ChartQA, InfoChartQA, DocVQA, InfoVQA, TATDQA). Across all 15 model-benchmark pairs, ECF improves average accuracy by 2.62 percentage points over the strongest fixed top-k input.
For LLaDA2.0-Uni, ECF versus other training-free options:
| Method | ChartQA | InfoVQA | TATDQA |
| Fixed top-2 | 24.60 | 30.12 | 11.52 |
| Answer-UQ top-1 | 30.72 | 36.08 | 14.08 |
| ECF | 31.12 | 42.60 | 18.96 |
With LLaDA2.0-Uni, ECF beats the strongest training-free competitor per dataset by 2.37 percentage points on average. Enlarging the candidate pool to k=4 and k=5 keeps macro accuracy stable at 30.19 and 30.32 (versus 30.11 at k=3), so the method does not break as the pool grows.
Retrieval augmentation is now standard for multimodal models, and diffusion language models are being pursued as a mainstream alternative to autoregressive generation. This paper pins down a structural clash between the two and offers a light, drop-in fix: rather than feeding everything, let the model's first-step entropy act as a gatekeeper. For RAG practitioners, ECF needs no retraining and no specific retriever, so it can be wired into existing pipelines directly.
The alignment of Theorem 1 is deliberately local: outside the correct-source basin, low entropy may flag a confidently wrong source, which is why retrieval rank must act as a complementary prior. The authors also stress that the causal construction used for comparison is an oracle isolating a dependence cost, not a universal ranking of model families. The DLMs tested are also relatively small (LLaDA2.0-Uni and peers are not on the scale of the largest autoregressive multimodal models today), so whether conflict is equally damaging on larger DLMs is left open.