NTU's DeltaTTG uses a Token-to-Go progress meter to boost multi-turn agent RL by up to 7.2 points

NanyangTechnologicalUniversity · hf · 2026-10-14

DeltaTTG: A reusable progress signal for RLVR with multi-turn agents

Problem: RL with verifiable rewards (RLVR) for multi-turn agents relies on sparse terminal rewards that reveal little about individual steps. Within-group step comparisons need reliable state matching and diverse sampled outcomes, while direct step scoring requires costly external evaluators, human annotation, or task-specific heuristics.

Method:

Introduces a shared progress meter learned from recorded trajectories of diverse policies with low-cost supervision and no task-specific progress rules
The supervision target is Token-to-Go (TTG) — the number of tokens remaining until task completion, an observable proxy for remaining effort
During actor training, changes in the frozen meter's estimates redistribute outcome-derived advantages across steps without altering the reward, preserving each trajectory's advantage sign and token-average scale

Results:

On ALFWorld, PaperSearchQA, and BFCLv3, DeltaTTG improves average score over GRPO by 5.7 and 7.2 points with Qwen3-4B and Llama-3.1-8B-Instruct respectively, beating all compared RL baselines on average
Actor-training GPU-hours stay close to GRPO
Further experiments show improved sample efficiency, out-of-domain performance, and the meter's ability to distinguish step quality and guide test-time action selection across actor sizes

Code: https://github.com/Xuyan923r/DeltaTTG

Original post →

NTU's DeltaTTG uses a Token-to-Go progress meter to boost multi-turn agent RL by up to 7.2 points

More from coding & agent