AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning
Zi-Han Wang, Zhengxi Lu, Zhiyuan Yao, Jinyang Wu, Jie Wu, Zhengzhou Cai, Yueqing Sun, Ziang Ye, Linji Hao, Qi Gu, Xunliang Cai, Yongliang Shen, Yujiu Yang
cs.AI, cs.LG
2026-08-06
GRPO averages trajectory success across every step, burying pivotal decisions; AgentOPSD aggregates teacher-student log-prob gaps into per-turn evidence and recursively updates a Bayesian belief, lifting a 7B agent to 89.1% on ALFWorld.
Reinforcement learning for agents with verifiable rewards (typically GRPO) gives one outcome reward for a whole trajectory, and that advantage is averaged across every token. In long-horizon multi-turn tasks the outcome is usually driven by a few pivotal decisions, and averaging the success across all of them buries those key steps in noise; the longer the task, the worse it gets.
Prior work brings privileged self-distillation into credit assignment for denser supervision, but how an isolated local signal should express sequential credit has stayed unclear.
AgentOPSD recasts this as a Bayesian belief-update problem. It needs no critic and no extra rollouts.
First, the teacher-student log-probability gap at each token is aggregated into per-turn evidence. The teacher is the same policy conditioned on success-relevant privileged information, the student is the standard policy, and their log-probability ratio is the evidence strength for that turn. Second, this evidence updates a belief about whether the trajectory will eventually succeed: a Bayesian belief is updated recursively in log-odds space, initialized from the group success rate. The same local evidence can be decisive while the outcome is open and redundant once the accumulated belief already points to an outcome, and the recursion captures exactly this history-dependent marginal revision. Third, the trajectory-level advantage is reshaped into per-turn advantages, signed, bounded, and direction-preserving, recovering standard GRPO at λ=0.
The whole mechanism is fully compatible with standard policy optimization; it only redistributes the learning signal across turns.
Evaluation spans three interactive environments and two model scales.
| Method (Qwen2.5-7B) | ALFWorld success |
| GRPO | 81.2 |
| Skill-GRPO (with privileged skills) | 88.3 |
| SDAR (strong self-distillation baseline) | 85.9 |
| AgentOPSD | 89.1 |
At 3B scale AgentOPSD reaches 84.4 on ALFWorld, 49.2 accuracy on Search-QA, and 90.2 score on WebShop, several of them best or tied. The ablation strips each component: dropping turn-level granularity to per-token accumulation falls to 85.9, replacing recursive belief with the raw local gap falls to 82.8, dropping the outcome-aligned signed direction falls to 80.5, and removing the success-rate prior anchor falls to 78.9. The signed direction and prior anchor matter most. λ=0.5 is best; smaller values degrade. A horizon-sensitivity plot shows AgentOPSD's success degrades far more gracefully than GRPO as the number of interaction turns grows.
It gives turn-level credit assignment a principled and cheap recipe: no critic, no extra rollouts, just recursive belief reshaping that turns sparse outcome reward into per-turn signal inside a standard RL loop. The theory links the self-distillation log-ratio to the ideal Bayes factor and explains why an isolated local gap is not, by itself, sequential credit. For long-horizon agent training this is a directly pluggable improvement.
The Bayes-factor approximation relies on a "success rate low enough" assumption (the ρk approaching zero limit in the paper). On tasks where success is not rare, the log-ratio estimate is biased, and behavior across a wider range of success rates is not validated.
Training needs success-relevant privileged skills to construct the teacher branch, and such privileged signals are not easy to obtain for every task. Against the strongest baseline SDAR, AgentOPSD is 3.2 points higher on ALFWorld-7B, but it is not a uniform win: WebShop accuracy of 79.7 is actually below SDAR's 82.8. Validation covers only three relatively small environments (ALFWorld, WebShop, Search-QA), and behavior at larger scale or on more open tasks is unknown.