Beyond Timestamps: Decision-Aligned On-Policy Distillation for Long-Horizon Agents
Mingju Chen, Can Lv, Jinrong Liu, Huan Zhang, Heng Chang, Shiji Zhou
cs.LG, cs.AI
2026-09-27
AlignOPSD scores student decisions in matched sibling contexts, then assigns span-level credit. Qwen2.5-3B/7B beats GRPO by 5.5-8.7% on eight agent aggregates, first in six.
Long-horizon language agents finish a goal through a chain of interdependent decisions, while the training signal is often a single terminal check. GRPO turns sibling-rollout outcomes into a relative advantage and broadcasts it across every response token. That sees whether the episode worked. It does not mark which mid-trajectory decisions helped, or where a failure started.
OPSD adds dense feedback from a privileged view of the same policy: the teacher sees extra train-time context such as a retrieved skill; at deployment that context is gone. Local scores still sit on the student's current timestamp.
Beihang and Tsinghua call this Decision-Timestamp Mismatch. Across rollouts, the same turn index often does not mark the same functional decision. On WebShop, 58.90% of same-turn student-student pairs already differ in action type. After entity-level matching (same product ID and so on), 79.72% of correspondences sit on different turns, median displacement 4. Inside a rollout, a decision such as comparing options often lasts several turns. Per-turn credit shards one decision; a fixed window mixes two.
Same-prefix probability comparisons remain valid. Using the local gap as a grade for the current functional decision does not.
AlignOPSD aligns supervision with functional decisions before it assigns credit. The teacher is not a larger model. It is the same πθ conditioned on train-only privileged information kx.
Supervision rectification uses thinking traces as a semantic stand-in for decision state, encodes them with frozen Qwen3-Embedding-0.6B, and scores cosine similarity across sibling rollouts of the same task. Candidates below threshold γH drop out; at most one source per sibling; at most top-K per target; plus an action-consistency filter (embedding cosine at least 0.8, and on WebShop matching operation type and click target).
The method does not imitate retrieved teacher text. The student's own sampled response au is teacher-forced under the matched privileged histories: log-probabilities are recomputed along the same tokens. The identity privileged view yields the usual local gap; aligned views mix in with confidence αu. No valid source means a fallback to the local gap. The scoring context changes. The optimized behavior does not.
Hierarchical assignment cuts variable 2-to-8-turn spans where Jensen-Shannon divergence between adjacent-turn correspondence profiles jumps. Trajectory advantage Aseq is the budget. Rectified teacher support that agrees with the outcome direction gets more of it: first across spans, then across turns inside a span, then broadcast to tokens. KL regularization keeps the allocation near uniform token mass; the token-weighted mean stays 1. There is no extra distillation loss. Correspondence, spans, and weights are stop-gradient.
Find the sibling context that is doing the same job, then spread the terminal win or loss over however many turns that job lasted.
Backbones are Qwen2.5-3B-Instruct and Qwen2.5-7B-Instruct. Tasks: ALFWorld success, Search-QA exact match, WebShop Score and Acc. Baselines cover prompting, GRPO-style RLVR, and OPSD plus RL methods including StepOPSD, SDAR, and RLSD. Training uses up to eight A800 GPUs on one node, 150 updates, eight sibling rollouts per task.
| Backbone | Metric | AlignOPSD | GRPO | StepOPSD |
| 3B | ALFWorld Avg | 80.5 | 75.0 | 73.4 |
| 3B | Search-QA Avg | 45.1 | 36.4 | 43.7 |
| 3B | WebShop Score / Acc | 86.8 / 68.8 | 79.8 / 63.3 | 82.4 / 66.4 |
| 7B | ALFWorld Avg | 89.1 | 81.2 | 88.4 |
| 7B | Search-QA Avg | 49.1 | 42.0 | 48.2 |
| 7B | WebShop Score / Acc | 87.9 / 78.9 | 80.9 / 72.6 | 87.2 / 78.1 |
AlignOPSD beats GRPO on all eight backbone-aggregate pairs by 5.5-8.7% and beats StepOPSD on all eight, ranking first in six. The two second-place finishes: GRPO+OPSD at 81.2 versus 80.5 on 3B ALFWorld, and Skill-GRPO (skills kept at validation) at 81.2 versus 78.9 on 7B WebShop Acc. Per-category numbers swing. On 7B ALFWorld Pick, AlignOPSD scores 70.8 against GRPO's 91.2.
A WebShop 7B ablation splits the two stages. Drop rectification and keep adaptive allocation: Score/Acc fall to 84.2/71.9. Keep rectification and replace spans with token, turn, or random allocation: 81.6/71.1, 86.3/77.3, 78.9/69.5. Turns already beat tokens; adaptive boundaries add another 1.6 points. Random spans are worst, so a wrong window hurts more than a coarse one.
Over the first 100 updates, off-turn match weight is 96.70%, 83.63%, and 22.71% on ALFWorld, WebShop, and Search-QA. Search-QA allows at most four turns, so there is little room to mismatch. Mean learned span length is 4.41, 3.92, and 2.24 turns. Credit temperature T=0.5 peaks in all three tested settings. K and γH do not: on WebShop 3B, K in {2,3,4} spreads endpoints by 11.7 points, and raising γH from 0.60 to 0.80 lifts success from 51.6% to 69.5%.
Dense supervision can be fine-grained and still grade the wrong decision. AlignOPSD keeps the backbone and the deployment interface, and only changes how a GRPO-style terminal advantage is carved. Code is public.
The useful setting is specific: multi-turn interaction, decisions that can be matched across sibling rollouts, and a verifiable terminal reward. Household control, shopping, and search QA all fit. Short trajectories where decisions already switch every turn leave little work for long spans; Search-QA gains are more likely from rectification.
For teams already running GRPO or OPSD, this is an incremental training-time fix. Extra cost is correspondence, re-scoring, and segmentation every batch. The paper counts that as training overhead and does not report wall-clock or extra forward passes. Hyperparameters do not transfer as a single set.
There is no standalone Limitations section.
The full ablation is WebShop 7B only. ALFWorld and Search-QA are not split the same way. Sensitivity shading is twice the local temporal variation, not across-seed error. Functional-span labels come from GPT-5.6 at aggregate resolution; there is no per-trajectory boundary F1, and the paper does not claim the learned spans recover semantic goals.
Thinking similarity is a proxy. On WebShop updates 1-100, same-state student-teacher cosine averages 0.7604, arbitrary cross-rollout pairs 0.7214, training-selected pairs 0.8805. Part of that last jump is the filter itself. The matcher covers 45.20% of target turns; the rest fall back to the local gap. Privileged context comes from the same SkillBank as skill-conditioned baselines. Privilege alone did not explain the gains. Retrieval quality is not swept.
Per-category swings mean an aggregate first place is not a first place everywhere. Compute for the alignment module is unreported. The teacher is a privileged view of the same policy; whether a stronger independent teacher changes the mismatch is untested.