LOPD learns what its teacher sees: 96 latent tokens beat GRPO with 30% of the rollout budget

Latent On-Policy Self-Distillation

Guibin Zhang, Jiayang Lyu, Ran Sun, Xinlei Yu, Haoyu Zhao, Qibing Ren, Shuicheng Yan

cs.LG, cs.CL

2026-08-13

LOPD turns retrieved experience into 96 learnable latent tokens for a frozen self-teacher, jointly trained with the student, beating GRPO with under 30% of its rollout budget.

What problem this solves

On-policy self-distillation (OPSD) trains an agent by letting a same-weight teacher, conditioned on extra privileged context such as a verified answer, a reference trajectory, or environment feedback, re-score every prefix the student actually visited and supervise it token by token. The signal is denser than outcome-reward RL, and no external stronger model is needed. The weak point is the context itself. Every existing method hands the teacher a designer-chosen artifact: OPSD injects an oracle trace, SDPO conditions on successful sibling rollouts, Skill-SD supplies a distilled skill summary. None of these formats is stable across settings. In the paper's own tables, SDPO on Qwen3-4B drops BFCL-v3 from 22.88 to 15.75 and ACEBench from 50.6 to 38.0, both below the untrained model; OPSD lands at 40.24 on LiveCodeBench against 45.61 vanilla. Adding privileged information made things worse, which points at the choice of context as the one variable nobody had optimized.

Method

LOPD makes the privileged context learnable and leaves the rest of the OPSD loop intact.

• An experience bank stores successful rollouts from the training split as task description plus action-result sequences, with verbose observations stripped. A Qwen3-Embedding-8B dense index retrieves the top-3 nearest entries for the current task.
• A composer encodes each task-experience pair with the frozen backbone plus a rank-8 LoRA, then a QFormer-style compressor (8 shared-weight cross-attention layers) squeezes it into 32 continuous latent tokens; three retrievals give 96 tokens total. Compression is task-conditional, so the same experience encodes differently per task.
• Injection uses a sentinel string in the prompt: after tokenization, placeholder embeddings are replaced with the latent tokens. This runs on unmodified SGLang/vLLM engines with normal KV cache and batching, and the paper verifies numerical equivalence between the token-ID path and the embedding-injection path.
• The teacher is a frozen copy of the base model, initialized identically to the student. It re-evaluates every student prefix with the latent context, and the student matches its top-20-logits-plus-tail distribution with reverse KL. At deployment only the student survives: no bank, no retriever, no composer.

The hard part is blocking a collapse shortcut. The composer can minimize the distillation loss by steering the teacher toward the student, which makes the context uninformative while the loss still goes down. LOPD stops this with a privileged-margin constraint: per action token, take the teacher log-prob minus the student log-prob, weight by trajectory outcome (success +1, failure −1), and require the average to stay above m=0.05, enforced by a dual variable that ratchets up on violation. An anchor term holds the latent context near its cold-start initialization, and the composer is first warmed up with next-token NLL on retrieved-experience-to-successful-trajectory pairs. The ablation shows this constraint decides the outcome: a frozen composer reaches 0.573 on EnvScaler; m=0 collapses to 0.551, below the frozen baseline, confirming the shortcut is real; m=0.05 reaches 0.637; m=0.20 slips back to 0.613.

Results

Best aggregate in all ten backbone-benchmark combinations across Qwen3-4B/8B and Olmo3-7B:

LOPD stays above the vanilla model in all ten settings, with gains from 1.50 points (BFCL-v3, Qwen3-8B) to 17.2 (EnvScaler, Qwen3-8B).

Sample efficiency is the headline. Over a 1,600-generation rollout budget, LOPD passes 0.61 mean reward at generation 320 and reaches its final 0.637 by generation 576; GRPO finishes the full budget at 0.611 and Skill-SD at 0.588. Capacity and retrieval both sit on thresholds: 8 or 16 latent tokens per experience stall near 0.56, 32 jumps to 0.637, and 64 or 128 add nothing consistent; one retrieval gives 0.605, three give 0.637, and more does not help.

The behavioral shift outlives training. The distilled student, running with no privileged context, takes more environment steps (17.04 vs 11.12) but makes far fewer tool calls per step (1.11 vs 3.50), shortens its first response by 37.5%, cuts repeated calls from 8.89 to 5.25, and raises reward per tool call from 0.038 to 0.050. The model switched from spraying speculative parallel calls to executing a sequential plan.

Why it matters

Three things carry over for practitioners. The deployed student has zero inference overhead: bank, retriever, and composer exist only during training, so the serving stack is unchanged, unlike memory or RAG schemes that push retrieval into the request path. The efficiency numbers matter wherever multi-turn rollouts are expensive. And the paper swaps the foundation under OPSD: fixed-format privileged context can backfire across settings, a learnable representation is the layer that stays stable, and the collapse ablation backs that claim directly. Within that framing it is an incremental but well-evidenced step: gains over GRPO mostly sit in the 1-5 point range, with the large jump (17.2) confined to tool use. Weights for Qwen3-8B-LOPD and Olmo3-7B-LOPD plus code are public.

Limitations

The authors concede the big one: projecting the latent tokens through the LM head yields fragmented multilingual and code-like mixtures, and task conditioning changes the surface without producing a readable procedure. Decodability does not establish what the teacher functionally uses, and the case study covers only ten tool-use and ten coding examples. Several margins look thin: +0.55 over the best baseline on EvalPlus with Olmo3-7B, +0.88 on BFCL-v3 with Qwen3-8B. The margin constraint needs a trajectory-level verifier (A=2r−1), so open-ended tasks without checkable outcomes are out of scope. The recipe carries three extra hyperparameter groups (margin m, anchor weight, dual step) on top of cold-start training, which is real reproduction cost. The teacher backbone never updates, so student progress cannot feed back into the teacher; the continual self-improvement framing in the introduction stays aspirational without cross-iteration experiments. Everything is validated at 4B-8B scale in two domains.

Terms

• OPSD (on-policy self-distillation): A same-model teacher conditioned on privileged information supervises the student's own rollouts token by token.
• Privileged context: Extra input only the teacher sees at training time; the student never receives it.
• Latent token: A continuous vector occupying a context slot like a special token, differentiable end to end.
• QFormer: A light module that cross-attends from learned queries to compress variable-length input into a fixed number of vectors.
• Reverse KL: The KL direction with the student distribution in front, concentrating the student on teacher-supported behavior.
• GRPO: Group-relative policy optimization, the RL baseline that normalizes advantages within a group of rollouts for one task.
• Privileged margin: A constraint requiring the teacher's log-prob advantage over the student to stay above a threshold, preventing teacher-student collapse.

Source

• arXiv 2608.13040v1
• PDF
• Hugging Face
• GitHub · https://github.com/bingreeky/LOPD

What people are saying

• LOPD: latent self-distillation beats GRPO using under 30% of its rollout budget burny_tech

Related papers

• Coding-agent residual streams linearly encode test outcomes 25 steps ahead
• Aletheia writes a publishable math paper unassisted; 13 of 700 Erdős solves are real
• AVO Makes the Coding Agent the Variation Operator, Beating FA4 by 10.5% on B200
• TinyCast emits a forecast distribution from 146K params by computing seasonality
• When models stay silent or lie, RelP beats black-box by 5 points; SAE does not
• Activation oracles and SAEs give no uplift on wild LLM bugs; CHIVE predictors match Opus

All paper explainers