RetrievalRouter: Joint Modality and Architecture Selection for Document Retrieval
Emre Kuru, Mehmet Onur Keskin, Reza Farahbakhsh, Noel Crespi
EMNLP 2026
cs.IR
2026-08-26
RetrievalRouter picks among five pipelines from the query text alone. At λ=0.1 it reaches 0.755 nDCG@5 in 0.666 s: 2.5% above multimodal late-interaction and 12.4× faster.
Document retrieval now splits on two axes. Modality: extracted text versus page screenshots. Architecture: one vector versus late interaction. The most accurate setup is also the slowest. Multimodal late interaction here is 8.283 s at 0.737 nDCG@5; BM25 is 0.019 s at 0.510. Production has to pick missed evidence or unusable latency, and it picks once for every query.
Across 11 financial, scientific, and open-domain sets, failures are query-level and asymmetric. Text pipelines drop on visually dense pages; visual encoders drop on long-range wording. No static pipeline wins everywhere.
The action set is five pipelines: BM25, text-dense, text-rerank, multimodal-dense, multimodal-rerank. Pure late interaction stays as a static baseline because rerank already recovers most of its accuracy at lower latency. The router is frozen Qwen3-0.6B-Base plus LoRA, mean-pooled to 1024-d, then a linear head. Overhead is 15 ms.
Labels are not hard winners. Each query gets a reward per arm: a convex mix of nDCG@5 and normalized latency. Sweeping λ traces the accuracy-latency front. Rewards go through a softmax at temperature 0.1, and the router is trained with KL to that distribution. Hard labels inject noise when several arms tie; soft targets keep ties as ties and stretch small gaps.
Text arms use Linq-Embed-Mistral and GTE-ModernColBERT, with Gemini 3.0 Flash captions on figures so missing OCR is not confused with architectural limits. Visual arms are Nomic Embed Multimodal and its ColPali-style counterpart.
At λ=0.1 the router reaches 0.755 nDCG@5 in 0.666 s. Versus ML that is +2.5% and 12.4× faster; versus MR, the usual deployable late-interaction stand-in, +3.0% and 1.7× faster. At λ=0.5 it is 0.707 in 0.314 s, +6.2% versus MD and slightly faster. At λ=0.7 versus Arabzadeh-style hard routing, 0.630 vs 0.624 nDCG and 0.148 vs 0.171 s, neither gap significant. Even at λ=0, 6.0% of queries go to BM25, 20.0% on Wiki-SS, so lexical match is sometimes the accurate arm, not a cheap fallback.
The oracle sits near 0.90 nDCG@5, about 14 points above the router. On the 1,828 queries where the oracle picks MD, MD's mean reward is 0.93 and adding rerank drops it to 0.70. As visual density rises, multimodal routing at λ=0 climbs from 69.7% to 95.2%.
A mixed corpus of filings, slides, and papers does not need ColPali on every query. A 0.6B router and one λ slide the same indexes along the accuracy-latency front. Code plus per-query best-arm labels on more than 80,000 queries are released.
This is routing, not a new retriever. Every arm still needs its index built up front.
Storage is the first bill: four vector indexes plus BM25. The multimodal late index is about 39 GB, 13× the text-dense index. Keeping every pipeline warm takes about 40 GB of VRAM. The trade is disk and cards for GPU time.
Splits are intra-dataset 80/10/10. The router may treat domain words as a proxy for visual density; zero-shot transfer is untested. Query text alone cannot decide "summarize the table on page 5" when the page layout is the missing bit. The action set has no generative LLM reranker. The 11 benchmarks are English-heavy financial and scientific pages.