EviBack: Search-Agent Reinforcement Learning via Evidence-Constrained Teacher Backoff
Xiao Ma, Zhiquan Hu, Yi Wei, Chenchen Zhao, Yijun Chen, Jicheng Zhao, Yuming Li Chuang Dai
cs.AI
2026-07-27
When the final answer is the only reward, about 69% of rollout groups are all-zero and give no comparative signal. EviBack adds a frozen Teacher (GLM-4.7-Flash) only to those groups, and beats Search-R1 on F1 across seven QA benchmarks.
Training multi-turn search agents with reinforcement learning, the Search-R1 line of work uses the final answer's correctness as the reward. That signal is sparse. Across 512 eight-rollout groups the authors counted, only 27% contain a contrast (one trajectory correct, the rest not); 69% are all-zero groups where RL learns no relative preference at all.
"Zero reward" also conflates three different failures: 64.5% are genuine evidence insufficiency, 16.7% are semantically right but with the wrong surface form, and 18.9% had enough evidence but a wrong or partial answer. Treating them as one failure type trains the model toward the wrong fix.
EviBack adds auxiliary supervision only to the all-zero groups, leaving groups with any verified hit on the Actor's own verifiable reward. The supervision comes from a frozen Teacher (GLM-4.7-Flash), run in two stages.
The hard constraint is that Stage B may not override Stage A's "I" judgment. If the evidence is insufficient, the Teacher cannot pretend an answer is possible just because the reference looks close. This gate separates evidence insufficiency from answer misalignment.
The two-stage Teacher itself is produced by an end-to-end automatic prompt engineering pipeline (E2E-APE, using GPT-5.5) that partitions, labels, ablates, and selects starting from a hand-written single-prompt version. It automates the painful job of hand-tuning a judge prompt.
Across seven open-domain QA benchmarks, three Qwen3 scales all compare against Search-R1.
| Scale | Search-R1 F1 | EviBack F1 | Gain |
| 0.6B | 0.2431 | 0.2490 | +0.0059 |
| 1.7B | 0.2509 | 0.2911 | +0.0402 |
| 4B | 0.4044 | 0.4177 | +0.0132 |
The 1.7B gain is the clearest (95% CI excludes zero); single-hop macro F1 rises from 0.3359 to 0.3985 and multi-hop from 0.1785 to 0.1975. Search behavior also cleans up: valid-answer rate goes from 0.73 to 0.86, mean searches drop from 1.73 to 1.59, and both duplicate queries and forced termination at max turns fall. The authors count 593 questions rescued from no-valid-answer or max-turn failures, against 140 regressions.
Anyone doing RL for agentic RAG hits the fact that all-zero groups dominate. EviBack offers a fix that preserves the verifiable-reward principle: the Teacher only backs up zero-reward groups and is entirely offline at inference. Splitting "is the evidence enough" from "is the answer right" is the most portable idea here.
The E2E-APE line, which automates Teacher construction, is a useful nudge for workflows that hand-tune every judge prompt.
The authors concede several points: they tried finer-grained signals (query quality, evidence redundancy, stopping confidence) and found no combination that gave stable gains, so the reward stays conservative; the train and eval sets are sampled subsets, so results do not generalize directly to the full datasets.
The most telling cost is at scale: at 4B, F1 rises but mean searches climb from 2.05 to 2.50 and duplicate-query rate from 13.7% to 24.8%, so Teacher intervention itself pushes up search volume, and the control cost grows with scale. The absolute F1 gains are modest (the 0.6B interval even crosses zero). The real contribution is behavioral, not leaderboard scores.