USTC & Alibaba's ADRS distills token-level credit for multi-turn agent RL, lifting ALFWorld success to 94.5%
量子位 · wechat · 2026-09-15
Researchers from USTC and Alibaba propose ADRS (Agentic RL with Self-Distilled Reward Shaping) to tackle the temporal credit assignment problem in multi-turn agent RL, where terminal rewards can't tell which intermediate token mattered.
Approach: during training, re-score generated tokens with a teacher context containing task-matched skills, convert teacher preference into reward-correlated token-level credit, and inject it before advantage construction. Rollout and inference remain skill-free.
Three layers of credit construction:
- Per-step centering and scale normalization of teacher scores;
- TVA gating: teacher signal passes only when teacher confidence correlates with actual returns within a comparison group;
- No separate distillation loss — gated token credit merges with environment rewards into the existing advantage operator.
Results (Qwen2.5-3B/7B, Qwen3-1.7B, 150-step budget):
- ALFWorld success 94.5% (+10.1pt over best baseline); 96.1% on 7B;
- WebShop Score/Success 87.5/76.6;
- Ablation: GiGPO 82.8% → no-TVA 87.5% → +TVA 89.8%;
- Action/style token update ratio hits 95.3, showing updates focus on decision-critical tokens;
- 60% of training data beats full-data GRPO; +11.9pt on unseen tasks.
Caveats: TVA uses batch statistics, not causal attribution; re-scoring adds training-time forward compute. Paper: arxiv.org/abs/2608.03223
More from coding & agent
- Tencent unveils Hyra 2.0, a research agent that builds reusable scientific tools — TencentHunyuan · 2026-11-02
- What broke when I ran a leaderless agent swarm with no orchestrator — 7 failure modes and fixes — LostHumanity67 · 2026-11-02
- Forced yes/no prompts push GPT and Claude to 95%+ confidence on Goldbach's unproved halt test — Nir777 · 2026-11-02
- Automation took over my daily work — so why am I still sitting in the office for 8 hours? — Am1rSori · 2026-11-02
- General Reasoning explains how to train agent swarms with RL and latency pressure — rosstaylor90 · 2026-11-02
- "Coding isn't the bottleneck anymore — usage windows are," says founder Kyle Gawley — kylegawley · 2026-11-02