One Size Does Not Fit All! Dynamic Retriever and Generator Selection for RAG
Neeraj Anand, Payel Santra, Partha Basuchowdhuri, Debasis Ganguly, Sumit Bhatia
cs.IR, cs.AI
2026-09-16
Stronger retrieval beats extra reasoning; the heaviest stack is not always best. Supervised DRAG hits 53.2 EM at 3.11s on HotpotQA vs 51.6 EM at 4.36s for the best static pipeline.
Most RAG systems pin one retriever and one generator for every query. Query difficulty is not uniform: some items are single-document factoids, others need bridging, composition, temporal or numeric constraints. Always running query decomposition, a cross-encoder, and high thinking burns latency on easy questions. Always using BM25 and short decoding starves hard ones. Retrieval adaptivity has mostly been "retrieve or not, and for how many rounds". Generation adaptivity has mostly tuned thinking on a fixed retriever. Joint assignment is underexplored.
This paper lays out a grid of retriever and generator complexities, then picks a pair per query.
Four retrievers, increasing cost:
Three open generator families expose test-time compute. GPT-OSS-20B has low/medium/high thinking; Qwen3-8B and Gemma-4-26B have a thinking switch. That is 12 configurations for GPT-OSS and 8 for each of the others. Context is always five passages.
Training-free DRAG-QPP is sequential. Average IDF on the corpus estimates query difficulty; high IDF (rarer terms) keeps a cheap retriever, low IDF escalates. Thresholds are the 75th, 50th, and 25th percentiles on train. After retrieval, base-model perplexity on the context decides the generator: low perplexity stays light, high perplexity adds thinking. Supervised DRAG-SFT fine-tunes Qwen3-4B-Instruct on efficiency-aware oracle labels and emits both choices in one decode. The oracle assigns the cheapest pair that answers correctly, falling back to F1≥0.8 if EM is not 1, else the most expensive pair.
Benchmarks are TriviaQA, HotpotQA, MuSiQue, with FRAMES as out-of-domain (routers trained on MuSiQue, tested as-is).
On the static grid, lifting retrieval from lexical to dense to rerank helps most on multi-hop sets. Extra-High often regresses: GPT-OSS High/High on HotpotQA is 51.6 EM, Extra-High/High is 48.8. Adding thinking under a fixed retriever is smaller and bumpier. On MuSiQue, weak retrieval moves EM only from 7.1 to 7.6; High retrieval moves 19.5 to 22.8. Reasoning does not repair bad retrieval.
The HotpotQA oracle distribution shows the same: many queries are already solved at low or medium cost. Latency runs from 0.11 s at Low/Low to 4.36 s at High/High and 5.87 s at Extra-High/High.
| Method (GPT-OSS, HotpotQA) | EM | F1 | s/query |
| Best static (RAGmax) | 51.6 | 65.2 | 4.36 |
| DRAG-QPP | 49.2 | 63.1 | 2.18 |
| DRAG-SFT | 53.2 | 67.6 | 3.11 |
| Oracle | 63.9 | 76.9 | 1.83 |
On TriviaQA, SFT reaches 68.1 EM versus RAGmax 66.1, at 5.37 s versus 10.73 s. MuSiQue is 24.5 versus 22.8. On FRAMES, SFT still beats RAGmax (23.3 vs 21.8), with weaker significance. Qwen and Gemma follow the pattern: QPP matches mid-to-high static quality at lower latency; SFT edges further. Routing overhead is 0.28 to 0.36 s; extra SFT latency is mostly from choosing heavier pairs.
Failures include overestimation (easy queries sent through decomposition, which injects noise), underestimation (hard queries under-provisioned), and entity mix-ups. Around k=5, strong retrievers already saturate.
For RAG capacity planning, moving retrieval from BM25 to dense plus a light rerank usually buys more than stretching thinking tokens. Extra-High decompose-and-rerank is not a free upper bound; it can lose. If fine-tuning is off the table, Avg-IDF plus context perplexity cuts latency while staying near a mid-high static stack. With oracle labels, SFT can win both quality and latency on HotpotQA.
This is scheduling over a configuration space, not a new retriever or a new reasoner. The oracle still sits about ten EM points higher, so routing is unsaturated.
QPP uses only Avg-IDF, a pre-retrieval signal, so QPP often trails RAGmax. SFT needs oracle labels from exhaustive runs of the same grid; new retrievers or models need relabeling. FRAMES gains are not always significant, so routing tracks query-type mix. Generators vary thinking, not model size; retrievers are the classic four rungs. k is fixed at 5, and latency tables omit index build. There is no head-to-head table against Self-RAG or Adaptive-RAG retrieve-or-not policies.