The Laws of Context Allocation: Causal Measurement and Closed-Loop Orchestration in Generative Search
Peiyang Liu, Xi Wang, Di Liang, Wei Ye
cs.LG, cs.CL, cs.IR
2026-08-24
A causal leave-one-out probe shows widening RAG context is an IR trap: the same 24 slots as 12 narrow rounds beat one wide pass by 14.4 portfolio-recall points.
Search engines hedge an underspecified query with a diversified ranked list. RAG keeps that premise and changes the delivery: retrieved passages go into one prompt, and the model compresses them into a single answer. On multi-facet questions that compression drops evidence-backed facts.
The design choice is how to spend a fixed inference budget. One wide context in a single pass, or the same number of document slots split across sequential narrow windows that rotate fresh evidence. Widening is the default in long-context NLP. Comparing the two packagings requires measuring which documents the generator actually used, not which documents look similar to the query.
The measuring tool is a causal leave-one-out (LOO) probe. After a response y is already generated, document d is deleted from the context and the drop in per-token log-likelihood of that fixed text is recorded. A larger drop means structural reliance. Because y is held fixed, each ablation is one teacher-forced forward pass, not a new autoregressive decode: k documents cost k+1 parallelizable forwards.
That probe exposes a diagnostic illusion. When the padding is off-query distractors on unrelated topics, query–document cosine and BM25 reach AUC near 1.000 at width k=24, while LOO sits at 0.829. Swap the padding for same-query hard negatives that share the topic but entail no gold answer, and BM25 / cosine fall to 0.444 and 0.484 at k=3, chance-level. LOO stays at 0.876 (0.824 at k=24). Similarity scores topical overlap. The probe scores generative dependence.
After calibrating away false positives from longer, more hedged free-generation text, width elasticity is −0.68 (SE 0.02): as the context grows, per-document attribution thins. This is not hardware truncation. Prompts peak at 5,485 tokens, well inside the model window.
The probe then closes the loop in a scheduler named Ascp. Retrieved documents are clustered into semantic facets. After each round the LOO signal discounts already-used facets through a monotone submodular objective, so the next window prefers unexplored evidence. A second lever, attribution-steered contrastive decoding, subtracts the over-used-document direction from the logits, with an adaptive-plausibility constraint keeping tokens on the model's manifold.
Baselines include feedback-free Rotate, Vanilla RAG, MMR, xQuAD, PM-2, DPP-RAG, and Carriage. Retrievers stay frozen on ALCE top-30 pools (a deep pool of 400 for the factorial). Generators are Qwen2.5-7B, Llama-3.1-8B, and Mistral-7B, with the allocation contrast repeated on Qwen2.5-14B/32B. Tasks are ASQA, QAMPARI, ELI5, plus a cross-cultural recipe rewrite as a non-English stress test.
The core experiment is a deconfounded k×T grid: widths {2,5,12,24} crossed with generation counts {1,5,12}. The rotation arm cycles disjoint rank windows. The fixed arm resamples the same top-k, isolating stochastic rereading from fresh evidence.
| packing | slots | rotation PR@T | fixed PR@T | fresh-evidence Δ |
| k=2, T=1 | 2 | 0.204 | 0.204 | 0 |
| k=2, T=12 | 24 | 0.397 | 0.257 | +0.140 |
| k=24, T=1 | 24 | 0.253 | 0.254 | 0 |
| k=24, T=12 | 288 | 0.421 | 0.336 | +0.085 |
Raising T from 1 to 12 lifts portfolio recall by 19.3–20.5 points at narrow widths (k=2: 0.204→0.397; k=5: 0.218→0.423) and still 16.8 points at k=24. That is the abstract's 16.8–20.5. Widening from 2 to 24 documents buys +0.050 on a single pass and only +0.024 at T=12. Beyond twelve slots the width gain is undetectable.
Matched at 24 physical slots, (2,12) beats (24,1) by 0.144 (95% CI [0.119, 0.170]). Fresh evidence accounts for 72% of the generation-count gain at narrow widths and 52% even at k=24. Twelve rereads of a fixed 24-document window ground on 10.3 documents. Rotation at (24,12) exposes 288 and grounds on 50.9.
Widening still helps on dispersed QAMPARI and hurts on head-concentrated ASQA (Llama at T=12: k=24 vs k=2 is −0.061). HotpotQA makes the mechanism obvious: a single pass must see both hop paragraphs at once, so widening gains +0.333; with twelve rounds the width effect flips to −0.046. Qwen2.5-7B-Instruct-1M at k=96 (about 11k prompt tokens) does not rescue width: on ASQA, k=48 vs k=24 drops 0.033. On 14B/32B the pooled (2,12) vs (24,1) gaps are +0.134 and +0.138.
On 2,400 paired held-out cells, Ascp reaches PR@T 0.309 against Carriage 0.276 (+0.033), PM-2-RAG 0.228 (+0.081), and Vanilla RAG 0.237. Deep rotation sits at 0.303, a +0.006 gap with BH q=0.343, not significant. Ascp wins on efficiency: ECR 0.626 vs 0.375, offering 10.55 documents against rotation's 25. Swap LOO for embedding similarity and the scheduling gain collapses to open-loop (−0.002, p=0.73). Steered decoding adds a further +0.0107 PR and +0.0555 ECR.
If the product goal is covering multi-intent or many-answer queries, a longer context window is not the default fix. Under a matched document budget, sequential narrow windows can rotate still-useful deeper ranks without spreading attention across a low-density tail. Open-loop diversifiers such as similarity MMR and xQuAD barely move the needle, because same-query candidates already live in one semantic neighborhood.
The bill is in Figure 6. On Qwen/ASQA, A100, probe removed, (2,12) vs (24,1) costs about 1.5× tokens and about 10× sequential latency (17.5 s vs 1.7 s). A 0.35 portfolio-recall floor is reachable with one wide pass. 0.40–0.45 is not, at any width in this grid. Test-time compute here is spent on rounds, not on window size.
Code and the probe ship with the paper. The practical shape is multiple short contexts plus a use-signal, not stuffing top-50 into one prompt.
There is no standalone Limitations section; the constraints sit in the setup. The main grid is 7–8B. 14B/32B only repeat the (2,12) vs (24,1) allocation contrast, not the full Ascp scheduler. Hyperparameters were frozen once on a 40-query ASQA development split. Retrievers are frozen, so the paper does not say whether a stronger first stage would rewrite the allocation rule.
The product assumption is that a user will consume a portfolio of responses rather than one synthesized answer. An integrate instruction and a structured single-response baseline that receives the T=12 decode-token budget still trail the portfolio by 2.9–10.9 points, with the gap shrinking as width grows. A one-answer product would claw some of that back.
Deep rotation nearly matches Ascp on PR. The closed loop sells token efficiency (ECR), not a higher coverage ceiling. The recipe task has almost flat relevance density (log-log slope −0.01), so wide windows are not punished by tail noise and rotation is the regime-correct policy. Calling −0.68 a "law" overreaches: on natural pools, redundancy strata span −0.43 to −0.67, and the protocol-clean −0.68 needs a fixed target and a width-invariant threshold. The endpoint is gold-answer-unit coverage, not whether anyone reads all T responses.