USTC & Alibaba's ADRS distills token-level credit for multi-turn agent RL, lifting ALFWorld success to 94.5%

量子位 · wechat · 2026-09-15

Researchers from USTC and Alibaba propose ADRS (Agentic RL with Self-Distilled Reward Shaping) to tackle the temporal credit assignment problem in multi-turn agent RL, where terminal rewards can't tell which intermediate token mattered.

Approach: during training, re-score generated tokens with a teacher context containing task-matched skills, convert teacher preference into reward-correlated token-level credit, and inject it before advantage construction. Rollout and inference remain skill-free.

Three layers of credit construction:

Per-step centering and scale normalization of teacher scores;
TVA gating: teacher signal passes only when teacher confidence correlates with actual returns within a comparison group;
No separate distillation loss — gated token credit merges with environment rewards into the existing advantage operator.

Results (Qwen2.5-3B/7B, Qwen3-1.7B, 150-step budget):

ALFWorld success 94.5% (+10.1pt over best baseline); 96.1% on 7B;
WebShop Score/Success 87.5/76.6;
Ablation: GiGPO 82.8% → no-TVA 87.5% → +TVA 89.8%;
Action/style token update ratio hits 95.3, showing updates focus on decision-critical tokens;
60% of training data beats full-data GRPO; +11.9pt on unseen tasks.

Caveats: TVA uses batch statistics, not causal attribution; re-scoring adds training-time forward compute. Paper: arxiv.org/abs/2608.03223

Original post →

USTC & Alibaba's ADRS distills token-level credit for multi-turn agent RL, lifting ALFWorld success to 94.5%

More from coding & agent