PCSD: Persistent Consistency Self-Distillation Tackles Sparse Rewards in Agentic RL

BIT · hf · 2026-08-05

LLM agents show strong potential in complex interactive tasks, but their reinforcement learning (RL) is often hindered by sparse rewards, where a long multi-turn trajectory may receive only a single outcome-level signal.

On-policy self-distillation (OPSD) provides dense token-level supervision, but the teacher model may not be reliable at every position. To address this, researchers proposed Persistent Consistency Self-Distillation (PCSD). This method derives token-level distillation weights from the local persistence of teacher-favoring signals. It combines adaptive windows with exponentially decayed aggregation, applies trend-aware modulation to attenuate declining support, and produces continuous weights via sigmoid gating.

Jointly optimized with GRPO, PCSD achieves the best ALFWorld Overall results among all baselines on both backbones without inference-time skills, exceeding GRPO by 15.6 and 13.3 points, and gaining 15.8 points over GRPO on unseen ALFWorld splits.

Original post →

PCSD: Persistent Consistency Self-Distillation Tackles Sparse Rewards in Agentic RL

More from coding & agent