When Privileged Guidance Misaligns: State-Matched Routing and Contextualized Self-Distillation for Multi-Turn Agents
Junzhuo Liu, Weiwei Li, Jun Ling, Peng Wang
cs.AI
2026-08-05
Privileged distillation can teach from a trajectory that no longer fits the state reached. SMRC-SD distills only at state-matched turns: Qwen3-1.7B rises 0.746 to 0.865 on ALFWorld, 0.574 to 0.693 on WebShop.
Training multi-turn agents (household tasks in ALFWorld, shopping in WebShop) has a long-standing issue: outcome rewards tell you only whether an episode succeeded, not which intermediate step was wrong. On-policy distillation (OPD) gives a denser signal by letting a teacher synchronized with the student re-score the student's per-turn response, with access to training-only "privileged" information such as a successful reference trajectory.
The direct approach (FullPath-SD) feeds the complete successful trajectory to the teacher at every turn. The paper isolates a flaw: in an interactive environment, the student's earlier actions keep changing the execution state (location, inventory, current page, completed subgoals). The student may take a different path or finish subgoals in a different order, landing in a state the reference never covered. The trajectory stays globally task-correct yet offers no executable next step from where the student actually is. The paper calls this state-reference mismatch. Forcing the teacher to re-score under such a mismatched reference can lower the probability of exactly the action token that was correct.
SMRC-SD treats a successful trajectory as a state-indexed resource rather than a blob of full text. Two stages:
State-matched routing. Each turn, an environment adapter builds compact state signatures: the reference side reconstructs a pre-action state at each position from its action prefix; the student side builds the current state from real history, observation, and the admissible-action set. The matcher checks three things: task identity (exact-task retrieval), state compatibility (an asymmetric relation |=g that lets the student have extra progress but not miss or contradict what the next step needs), and candidate admissibility (the reference's next action grounds into the current admissible set). A match is flagged, and the latest matching position is chosen to avoid repeating completed subgoals. Distillation applies only on matched turns; other turns run GRPO alone.
Contextualized self-distillation. For a matched turn, the teacher context ct is rendered from three fields: the complete successful path (global structure), a one-line current-state summary (where we are), and the grounded candidate next action (local continuation). The teacher re-scores the student's original sampled response under that context.
The final loss keeps GRPO on all turns and adds an SDL term only on matched turns (λSDL=0.01). Crucially, every privileged component (reference, signature, match, candidate, teacher context) is training-only; at inference the student uses only the ordinary prompt.
Two complementary benchmarks: ALFWorld (embodied household, TextWorld) and WebShop (web shopping).
| Model / benchmark | FullPath-SD | SMRC-SD |
| Qwen3-1.7B ALFWorld Avg@4 | 0.746 | 0.865 |
| Qwen3-1.7B ALFWorld Pass@4 | 0.836 | 0.914 |
| Qwen3-1.7B WebShop Acc | 0.574 | 0.693 |
| Qwen2.5-3B ALFWorld Avg@4 | 0.766 | 0.883 |
Baselines: on Qwen3-1.7B, plain GRPO scores 0.717 on ALFWorld and SDAR 0.578; SMRC-SD's 0.865 beats both, and also beats the 0.844 SDAR reports in its own paper.
Ablations separate the two designs cleanly:
A fixed-state teacher intervention measures the harm directly: the same FullPath intervention shifts teacher preference toward the candidate by +1.158 on matched turns, but lowers the already-agreeing action's score on unmatched turns (mean −0.052). Mismatched references actively hurt; they are not merely useless.
An engineering bonus: SMRC-SD produces shorter, less repetitive responses, averaging 78.8 tokens (close to GRPO's 79.5) versus 142.6 for FullPath-SD and 255.6 for Skill-SD; 4-gram repetition is 8.8% versus 20.7% and 38.6%.
The paper flags an upstream problem in multi-turn agent distillation that is easy to overlook: a successful trajectory has local validity. Each of its steps is responsible only for the pre-action state in which it was demonstrated. Once the student's trajectory forks, the whole trajectory stops being an unconditionally teachable script. Making "treat trajectories as conditional plans, verify local validity before use" an explicit check gets closer to the root cause than methods that weight or mask the distillation signal after the fact.
For anyone training agents: if you distill from successful trajectories (many RL agent pipelines do), first ask whether the student's current state still connects to that trajectory. If not, skip that turn and let GRPO handle it. The cost is writing a state-signature adapter per environment, which is the main engineering gate.
The authors concede:
A concern from reading: the evaluation covers only ALFWorld and WebShop, two relatively structured, discrete-state environments where signatures are tractable. Whether hand-built signatures hold up in continuous-state, high-dimensional-observation settings (real browser control, complex tool use) is unaddressed. All reference trajectories come from ready-made successful demonstrations (ALFWorld expert walkthroughs, WebShop traces built from the target product), so the method assumes high-quality references exist; without them it falls back to plain GRPO. The gains are consistent, but two models and two benchmarks is a modest surface.