DocMemo: Dynamic Evidence Discovery via Probabilistic Memory-Guided Retrieval for Multi-Modal Document Understanding
Hanshu Yao, Janfeng Zhong, Niu Lian, Jinpeng Wang
cs.AI, cs.CL, cs.IR, cs.MM
2026-08-07
DocMemo adds three memory levels and Bayesian page selection with Thompson sampling to long-document QA, hitting 77.6 average across three benchmarks and +15 points on PaperTab in 1.24 rounds.
Multimodal long-document QA, say asking for a specific number in a hundred-page financial report or paper, must find sparse, scattered evidence across text, tables, and figures. The mainstream approach retrieves a fixed top-k page set up front and then answers, known as static retrieval. Its flaw is that once the early pages are wrong, there is no recovery.
Recent iterative methods allow multiple retrieval rounds, but they usually rebuild the context each round with no real cross-round state propagation, effectively repeating independent retrieval and unable to track how page relevance shifts across rounds. DocMemo reframes long-document reasoning as dynamic evidence exploration, giving the retrieval process a state that updates and remembers.
The core is a three-level retrieval memory that separates persistent from dynamic information.
At inference, Thompson sampling draws θi from each page's Beta distribution and scores candidates as scorei = (1−λt)ci + λt·θi, with a λ schedule of [0, 0.3, 0.6, 0.6] that leans more on the posterior over rounds. Two mechanisms matter: spatial proximity propagation (when a page is judged relevant, neighbors get α += γ^|j−i|, with r=2, γ=0.5, motivated by the spatial contiguity principle in cognitive load theory), and adaptive granularity (page-level crops for dense regions, e.g. up to 5 table crops, images capped at 1500px longest side) to offset detail lost in full-page representation.
Three benchmarks: MMLongBench-Doc (1082 questions, 135 documents, up to 112 pages), LongDocURL (2325 questions, 396 PDFs), PaperTab (393 questions, 307 papers). GPT-4.1 serves as the binary judge.
| Method | MMLong | LongDoc | PaperTab | Avg |
| SimpleDoc | 60.6 | 72.3 | 65.4 | 66.1 |
| DocMemo | 71.3 | 81.1 | 80.4 | 77.6 |
DocMemo beats the next best by 3.7 and 8.8 points on MMLong and LongDoc, and by 15.0 on PaperTab. It averages only 1.24 rounds (SimpleDoc is fixed at 3), at 0.41 relative compute cost and roughly 2.4x efficiency. Thompson sampling scores 71.28 versus 68.62 for greedy selection. Ablations show that removing page belief memory or Bayesian updating drops the score to 68.80; OCR-only (no vision) drops to 58.6 and vision-only to 69.3, so both modalities are necessary.
Long-document QA is a core capability for document intelligence and agentic retrieval, and "retrieval that cannot correct itself dynamically" is a common pain point. DocMemo models retrieval as belief-state exploration, balancing exploiting found evidence against exploring new pages via Bayesian updates and Thompson sampling. For RAG and agent builders, the three-level memory split (schema, page belief, episodic) and spatial proximity propagation are directly borrowable, and the fact that it scores higher with fewer rounds shows that dynamic retrieval is about carrying state, not running more rounds.
The paper has no dedicated limitations section, but a few show through: the third iteration brings clearly diminishing returns (per-round accuracy drops from 72.0 in round 1 to 46.2 in round 3); offline preprocessing costs 1.2GB of visual-embedding storage and about 0.51 hours per 10 documents; full-page representation loses detail (mitigated by adaptive-granularity crops, though the crop strategy is coarse, with the 5-table-crop cap set by hand). All answer judging relies on GPT-4.1, and the bias of the judge model itself is not discussed.