VLM evidence attribution improves with quote-and-retrieve, not bounding boxes

Zhuchenyang Liu · hf · 2026-07-28

The paper studies how to improve evidence attribution in visual document understanding without relying on coordinates or region labels.

On a verified bilingual CiteVQA subset, the authors compare two interfaces:
a coordinate interface, where the model outputs bounding boxes;
a language interface, where the model quotes evidence in text and a retriever maps quotes back to page regions.
Across six open VLMs, the language interface raises evidence recall from at most 8 points to 26–47, while roughly halving hallucination, with little change in answer quality.
They then turn the quote-and-retrieve setup into a training scaffold.
Using GRPO and a reward based on the gold answer plus crops of retrieved regions, they train a model to quote better evidence without region labels.
On an 8B backbone, strict attributed accuracy improves from 22.4 to 33.8.
The takeaway: evidence attribution can improve materially even without a coordinate interface or expensive region-level supervision.

More from Multimodal