EviSD: Evidence-Conditioned Self-Distillation for Search-Augmented Agents
Jianan Xie, Xin Sun, Zhongqi Chen, Xing Zheng, Shu Wu, Bowen Song, Liang Wang
cs.CL
2026-08-03
EviSD refines RL credit on search and answer tokens using privileged evidence and gold answers, beating the best baseline by 1.3-2.3 EM points across seven QA benchmarks.
Search-augmented language agents learn through multi-turn interaction: they interleave reasoning with calls to a retriever, deciding when to search, what to query, and when they have enough evidence to answer. Training these agents with reinforcement learning usually means scoring the whole trajectory by whether the final answer was correct, then spreading that one reward evenly across every generated token (GRPO turns it into a group-relative advantage first). The problem is a trajectory often contains several searches of wildly different quality, one query might surface the decisive passage, another might be redundant or off-target, but outcome-only credit can't tell them apart. A correct answer might follow from a lucky guess as easily as from a good search strategy. Prior fixes either add hand-designed process rewards or use on-policy self-distillation, where the same model re-scores its own rollout under a privileged context and distills the difference back into training, but these either change the underlying training objective or smear the correction across the entire response instead of pinpointing the actions that actually mattered.
EviSD treats information already present in the training data, per-instance supporting evidence and gold answers, as privileged context. During training, the student samples a search or answer action from the ordinary context; the same model then re-scores that exact sampled action under a teacher context that includes the evidence (for search actions) or the gold answer (for answer actions). The gap between the two log-probabilities is the self-distillation signal, and it's applied only to the tokens of that generated search query or answer span; everything else, reasoning tokens, retrieved passages, keeps its original GRPO advantage untouched. The signal is squashed through a bounded tanh function and added on top of the outcome-derived advantage: when the teacher agrees with the sampled action, its credit is strengthened; when the teacher disagrees, it's softened, but the correction never flips the sign of the original outcome reward. At inference time none of this privileged context exists; the agent behaves like an ordinary search agent.
Across seven QA benchmarks (three single-hop, four multi-hop) and three backbones spanning different scales and generations, EviSD posts the highest macro-average exact match every time, beating the strongest compared method by 1.3-2.3 points. It does this while modulating only 6.7-15.1% of response tokens, evidence that the correction is genuinely localized rather than broadly re-weighting the whole response. Ablations back up both design choices: giving privileged context to answer actions only (dropping the evidence-based search privilege) costs 2.1 average EM points, and replacing the localized modulation with a response-wide auxiliary distillation loss (the SDAR-style alternative) costs 7.1 points, with the biggest drop on multi-hop tasks. During training, EviSD reaches the same success rate using only 1.87 searches per trajectory, versus 2.16 for SDAR and 2.27 for RLSD, indicating it teaches more informative queries rather than more retrieval.
This addresses a concrete engineering gap in RL for search agents: how to use the fine-grained evidence annotations that training sets already contain to sharpen credit assignment, without touching the outcome reward that anchors the update direction, without adding an auxiliary loss, and without any inference-time cost. Teams already training search agents with GRPO-style methods can plug this in directly wherever per-instance evidence annotations exist, no reward redesign required.
The method depends on datasets that already ship fine-grained supporting evidence, grouped by source document; without that annotation it doesn't apply. The benchmarks are all QA retrieval tasks where search and answer actions are cleanly separated; the paper doesn't discuss how the design would extend to broader agentic settings with more varied action types, where deciding which action gets which privileged source is less obvious. The modulation hyperparameters (sharpness and correction cap) are tuned on Qwen2.5-7B and carried over unchanged to the 3B and 1.7B models; the paper acknowledges it didn't separately verify whether smaller models need their own tuning.