Teach the Magnitude, Not the Direction: Verifier-Bounded Credit Assignment for Multi-Turn Multi-step LLM Agents
Zechuan Wang, Siyuan Lu, Hongxuan Zhang, Linjian Mo, Chenyi Zhuang, Leilei Gan
cs.AI
2026-08-13
CrEST factorizes per-token advantage into turn-segmented verified advantages times entropy-gated self-teacher modulation, reaching 52.0% average on BFCL V3, ahead of GRPO and all distillation baselines.
RLVR trains tool-use agents on rewards a verifier can check, which keeps the reward clean. But standard GRPO scores one full session trajectory and broadcasts that single number to every token. Within a single turn this is just noise. In a multi-turn session each turn has its own outcome, yet all turns share one reward: when turn 1 succeeds and turn 2 fails, the tokens of the failed turn still get reinforced. The credit assignment is structurally wrong. On WildToolBench no model out of 57 LLMs exceeds 15% session accuracy, and performance degrades sharply with turn count.
On-policy distillation (OPD) looks like the natural fix: a teacher scores every token, so credit assignment is dense by construction. But it requires a same-family, tokenizer-matched teacher and caps the student at the teacher's level. Self-distillation (OPSD) drops the external teacher by conditioning the student on ground truth, and then collapses within roughly 100 steps because gradient mass concentrates on a few tokens.
CrEST (Hierarchical Credit Assignment via Entropy-Gated Self-Teacher) factorizes the per-token advantage into two terms: a turn-level advantage that sets the direction, and a token-level modulation that only scales the magnitude.
The turn level is straightforward. Each turn gets its own verified reward from the environment, and group-relative advantages are computed independently per turn within the GRPO rollout group. A failed turn receives a negative advantage regardless of what the other turns in the same trajectory did.
The token level is where the design lives. The teacher is still the student itself, but conditioned on the ground-truth answer as privileged context; for each token it computes the log-probability gap between teacher and student. This gap never acts as an advantage on its own. It only modulates magnitude, multiplied by λ, and can never flip a sign. Two gates control it:
The final per-token advantage stays bounded, with a bias of at most 8.4% relative to the GRPO gradient at λ=0.3. CrEST adds exactly one tunable hyperparameter (λ) over standard GRPO; the other constants are fixed across all experiments.
| Method | BFCL V3 avg (4B) | WildToolBench session (8B) |
| GRPO | 43.63 | 5.47 |
| MT-GRPO | 49.25 | 7.03 |
| EnvTuning | 47.25 | 7.81 |
| OPD | 44.50 | 3.91 |
| OPSD | 38.75 | 3.91 |
| CrEST | 52.00 | 9.38 |
Across two model scales (Qwen3-4B-Instruct, Qwen3-8B) and 14 evaluation cells, CrEST takes the best or tied-best result in 13. Gains are largest exactly where credit assignment hurts most: on BFCL Long Context, the longest-trajectory split, it beats the strongest baseline by 7.0 points (4B) and 2.0 (8B).
The ablations separate the two levels cleanly: turn segmentation alone moves 43.63 to 47.88, token modulation alone reaches 48.75, and the combination reaches 52.00. Neither half approaches the full model. Removing the direction gate drops it to 46.75, removing the entropy gate to 46.25, removing both falls back to GRPO-level 43.50.
One training-dynamics measurement is worth a look. For OPSD the top-1% of tokens carry 42% of total gradient and the top-5% carry 77%, so a handful of tokens dominate the update. GRPO is diffuse (top-10% ≈ 31%). CrEST sits in between (top-10% ≈ 57%): differentiated, but not collapsed.
Agent training faces a real dilemma: RL has a clean ceiling but sparse signal, distillation has dense signal but a capped one. This paper's answer is directly usable. No external teacher, no reward engineering, just one extra privileged-context forward pass and two gates on top of a standard GRPO loop. Because the verifier decides every gradient direction, the student can in principle exceed the teacher, and the ablations show it does in practice.
For teams already running multi-turn agent RL, the turn-segmented advantage half can be lifted on its own: it needs no teacher signal at all and costs almost nothing to implement.
The authors' own list: validation covers two tool-use benchmarks at 4B and 8B only, and the entropy gate uses surprisal as a proxy for token importance, which works for tool-call trajectories but may not generalize to other generation contexts. The turn segmentation also relies on the assumption that all rollouts in a group share the same fixed sequence of user turns, which does not extend to variable turn structures, terminal-only rewards, or strongly coupled turns. That restriction sits in the verification structure itself, not just in benchmark choice.
On the numbers, WildToolBench session accuracy moves from 7.81 to 9.38; every method remains in the single digits, so this battlefield is far from solved. Each method uses a single training seed, and the three evaluation decodes at near-zero temperature measure decoding consistency, not training variance.