Less can be More: Relieving RAG Bottlenecks via Evidence Frontloading and Pressure-Adaptive Budgeting
Weibin Cai, Reza Zafarani
cs.CL, cs.IR
2026-08-26
PACE frontloads multi-hop evidence, then shrinks rerank budget under queue pressure. On HotpotQA, D=20 yields 60.26% complete recall@5 versus 41.13% at D=100.
Most RAG efficiency work targets the generator: compress the context, reuse the KV cache, schedule prefills. That is the wrong bottleneck under load. Cai and Zafarani run an open-loop serving simulation and compare p95 queue time at the reranker versus the LLM. Raise QPS or the reranking budget D, and the cross-encoder becomes the limiter even when it is far smaller than the generator. Qwen2.5-3B with MiniLM has a 92.5x parameter ratio; the pair is generation-heavy at low QPS and ranking-heavy by QPS 2.5. With Qwen-3B and DeBERTa, D=100 makes reranker queueing dominate end-to-end latency once QPS hits 1; drop D to 50 and the LLM is the limiter again. Context compression shortens the LLM queue. It does nothing for requests already waiting on the reranker.
A small fixed D leaks multi-hop evidence. Answering those questions needs several complementary passages, and diversity rerankers such as MMR and Dartboard penalize documents that look similar yet must stay together.
PACE (Prioritized Adaptive Coverage of Evidence) is training-free. It reorders an existing candidate pool, then decides how many of those candidates the reranker may score.
Evidence frontloading treats selection as covering query semantics with as few documents as possible. On each semantic dimension the selected set keeps a max over weighted document values; a new document only scores for uncovered mass. Direct query-document scores supply one weight. Soft anchors supply another: passages similar to highly relevant seeds are promoted, which is how bridge facts enter the prefix. The objective is monotone submodular, so greedy selection has a (1-1/e) guarantee. They retrieve with splade-v3, whose non-negative representations match the coverage math.
Pressure-adaptive budgeting watches both queues. If the reranker backlog is no worse than the LLM backlog, the query keeps Dmax=100. If the reranker is more congested, D drops in whole batches down to Dmin=20. The rule spends the largest budget that does not make ranking the bottleneck.
Offline evaluation uses HotpotQA (1,087 queries), MuSiQue (2,317), and 2WikiMultiHopQA (2,861). HotpotQA and 2Wiki keep only queries whose gold evidence already sits in the retriever's top-100, so the comparison isolates ranking rather than first-stage misses.
Under a fixed budget, PACE moves complete evidence earlier. On HotpotQA, standard dense retrieval at D=100 yields 41.13% complete evidence recall after reranking to K=5. PACE at D=20 reaches 60.26%; at D=50 it reaches 62.38%. Dartboard at D=20 falls to 33.22%. On 2Wiki the full-budget baseline is 48.76% and PACE at D=20 is 53.94%. In the online HotpotQA run at QPS=1.8, PACE selects roughly half of D=100 yet posts about 20 points higher recall@5 after reranking. A smaller, denser candidate list is easier for the reranker to promote into the final K=5.
This is a serving patch, not a new retriever. It leaves the cross-encoder untouched and skips iterative multi-hop search. For ranking-heavy RAG, packing evidence into the prefix and shrinking D under queue pressure is more on-target than compressing the generator's input. The submodular bound is the difference from another tuned heuristic. Stacks already using SPLADE plus a cross-encoder can insert the reorder step before reranking.
The paper measures whether evidence reaches top-K, not whether the answer string is correct. Incremental, and specific to ranking-heavy load.
There is no dedicated limitations section. The headline metric is evidence recall; end-to-end QA accuracy is not given the same status. Filtering HotpotQA and 2Wiki to queries whose evidence is already in the top-100 assumes a competent first-stage retriever. The serving study runs on a Quadro RTX 6000 with about 22 GB, Qwen2.5-3B, natural EOS, and a 128-token cap. Coverage needs non-negative encodings; a dot-product dense retriever is untested. Throughput-based pressure estimates may not hold under bursty arrivals or a faster reranker.