Predicting Partial Answer Quality and Utility in Agentic Retrieval-Augmented Generation
Fangzheng Tian, Debasis Ganguly, Craig Macdonald
cs.IR
2026-09-15
Probing Search-R1 and R1-Searcher after each loop shows sparse decisive steps. Quality prediction hits Pearson r=0.438; early stopping cuts ~11% of loops at ~98% F1.
Agentic RAG lets a model retrieve repeatedly while it reasons. Training usually watches only the final answer and the format. Whether a given retrieval-reasoning loop raised the answer, hurt it, or did nothing stays hidden. Trajectories often keep going after quality has already plateaued, burning compute and sometimes dragging the answer down.
The University of Glasgow probes the model after every loop: insert an answer trigger, greedily decode the current best answer, score it against gold with F1 as partial answer quality P, and take the adjacent difference as partial utility U.
A trajectory is the question, the initial reasoning, and each loop's query, retrieved context, and reasoning. Probing does not rewrite the main trace; it only asks for the most likely answer at a cut. At i=0 that answer is parametric knowledge plus the opening thought.
Two prediction tasks estimate P and U from the prefix so far. Intra-iteration signals describe the current query, context, reasoning, and answer confidence. Inter-iteration signals track how retrieval lists and reasoning change. Question-iteration alignment tracks drift from the original question. Unsupervised features include NQC, semantic concentration, RBO, and SBERT cosine. Supervised heads are all-MiniLM-L6-v2 cross-encoder regressors over structured text. Optional probing adds length-normalized generation probability of the probed answer. Those scores from a window of the last w loops, w in 1 to 5, feed a two-layer MLP with 16 and 8 hidden units.
Early stopping slices predicted P and U into four states. High quality and high utility stop now. High quality and low utility is a plateau. Low quality and high utility continues. If the trace once reached high quality and later falls into both-low, it rolls back to the earliest high-quality answer; iteration cost still counts through the rollback trigger.
Both pipelines sit on Qwen2.5-7B. Search-R1 is single-stage outcome RL on HotpotQA and NQ. R1-Searcher is two-stage RL on HotpotQA and 2Wiki. Retrieval is E5 top-3 from a 2018 Wikipedia dump. Dev sizes: HotpotQA 7405, 2Wiki 12576, MuSiQue 2417.
Search-R1 on HotpotQA ends at F1 0.549, ΔF1 0.265 versus the pre-retrieval probe, mean per-iteration utility 0.097, mean length 2.730; on 2Wiki, F1 0.429 and length 3.463. R1-Searcher is shorter and sharper: HotpotQA F1 0.529, ΔF1 0.408, length 2.311. On both systems U spikes around zero. Most loops barely move F1; a few swing hard. Positive-net traces jump early then plateau or dip. Negative-net traces drop early and rarely recover.
| Pipeline | Best quality r / τ | Best utility r / τ |
| Search-R1 | 0.438 / 0.345 | 0.321 / 0.208 |
| R1-Searcher | 0.401 / 0.320 | 0.321 / 0.250 |
Quality is the easier target because U is a difference. Supervised heads carry most of the signal. Unsupervised features help more with larger windows. Probe confidence helps and costs an extra decode. On Search-R1, θP=0.3 and θU=0.2 cut mean iterations from 3.21 to 2.86 (10.89% fewer) while keeping 97.60% of natural-stop quality; θU=0.3 keeps 98.49%. Fixed caps save steps too, with a steeper quality drop.
If you ship a Search-R1-style model that decides when to search again, its native stop rule is not aligned with answer quality. Measuring P and U makes that mismatch visible. A cheap trajectory-based heuristic then trims about a tenth of the loops without giving up much F1. Pearson r around 0.44 is not a precision controller. It is enough for a process switch.
The study stays on a single-trace agentic loop without self-reflection or document refinement. F1 on short answers will miss finer changes in long-form output. Stopping is a threshold heuristic, not a learned policy. The supervised regressors train on the pooled training splits of the three datasets and will need a refit out of domain. Probing has generation cost; dropping it at inference weakens quality prediction. The predictors are not wired into path selection or multi-agent retrieval.