PCSD: Persistent Consistency for Self-Distillation in Agentic Reinforcement Learning
Chunji Lv, Yangguang Wei, Junlin Liu, Yang Gao, Ming Liu, Xinming Wang, Jinyang Wu, Guoren Wang, Changsheng Li
cs.AI
2026-08-03
PCSD picks credible teacher signals for agentic-RL self-distillation by local persistence, trained with GRPO; best ALFWorld Overall on two backbones, beating GRPO by 15.6/13.3 and SDAR by 6.2/5.5.
Reinforcement learning for LLM agents is bottlenecked by reward sparsity: a trajectory spanning dozens of turns and hundreds of tokens often yields a single success or failure signal at the end, leaving it unclear which intermediate actions earned credit or blame. Critic-free methods like GRPO scale well but stay coarse. On-policy self-distillation (OPSD) adds dense per-token supervision from a frozen teacher with privileged skills, but the teacher is not reliable at every position, and indiscriminate distillation passes on wrong preferences too. Existing credibility methods either estimate token-level discrepancy in isolation (noise-sensitive) or assign a shared step-level weight (too coarse to capture positional variation). PCSD balances the two.
The key insight: informative teacher advantage should persist across a local neighborhood, while isolated spikes are more likely sampling noise. PCSD turns this into token-level distillation weights:
This weighted distillation objective is optimized jointly with GRPO (L = LGRPO plus lambda times LPCSD, lambda=0.01). Only the student updates; the teacher is frozen and scores the same student tokens by teacher forcing, with no resampling.
ALFWorld gives the best Overall on both backbones:
| Backbone | PCSD | GRPO | SDAR |
| Qwen2.5-3B | 90.6% | 75.0% | 84.4% |
| Qwen3-1.7B | 59.4% | 46.1% | 53.9% |
That is 15.6/13.3 points over GRPO and 6.2/5.5 over SDAR. On WebShop, Qwen2.5-3B has the top Score of 85.0 and ties SDAR for the top Acc; Qwen3-1.7B scores 78.9, second only to the Skill-GRPO star variant. A key point: PCSD needs no skill retrieval at inference, so the score comes purely from the trained student. Generalizing to the unseen ALFWorld split, Overall reaches 86.7%, well above GRPO at 70.9% and SDAR at 72.7% (plus 15.8 over GRPO). Ablations show adaptive aggregation, trend modulation, and exponential decay are all necessary (dropping to 82.8/83.6/85.1 without them); lambda=0.01 is best (0 gives 75.0, 0.005 gives 87.5, 0.05 gives 83.6). In training dynamics, 20% to 28% of tokens carry a weight above 0.5, concentrated rather than hard-selected.
It gives an interpretable answer to unreliable teachers in self-distillation: do not look at pointwise discrepancy, look at whether support persists locally. And because no skill retrieval is needed at inference, skills are internalized into the student during training, making deployment cleaner. For teams doing agentic RL who want dense supervision to patch sparse rewards, this weighting rule can be bolted onto an existing GRPO pipeline.
The authors concede in the discussion that aggregation and gating use fixed hyperparameters and the teacher is frozen, which isolates the effect of persistence weighting at the cost of adapting to evolving trajectory statistics and teacher reliability. A few more concerns: evaluation covers only two text-simulated environments (ALFWorld household tasks, WebShop shopping), with no real-GUI or complex tool-use agent benchmarks; backbones are only 1.7B/3B small models, and Qwen3-1.7B's absolute success rate of 59.4% is still low; the whole method depends on a teacher with a privileged skill library, and skill retrieval is just keyword matching, fairly crude, so teacher strength caps the ceiling; running a frozen teacher forward roughly doubles training cost.