Sequence-level OPD learns endings first; benign refusal goes from 17.5% to nearly all

Which Tokens, in What Order, and How Fast? A Theoretical Analysis of On-Policy Distillation Objectives

Hubert Strauss, Liam Fowl, Noam Razin

cs.LG, cs.AI, stat.ML

2026-11-18

Sequence-level on-policy distillation learns endings first and lifts benign refusal from 17.5% to nearly all; forward KL raises rare tokens exponentially faster than reverse KL.

What problem this solves

On-policy distillation (OPD) trains a student on its own samples so those samples move toward the teacher. Labs already use it to compress models, merge experts, and self-improve from feedback. Off-policy distillation copies sentences the teacher wrote, so training conditions on the teacher's prefix and test time conditions on the student's. OPD removes that mismatch.

The unsettled choice is the objective. Granularity is the full sequence, or the next token at each prefix. The divergence is usually forward KL (FKL), reverse KL (RKL), or Jensen-Shannon (JS). The usual rule only separates covering every teacher mode from sticking to high-probability modes. It does not say which tokens move first, in what order, or whether the gradient is still alive when the student is far from the teacher. Princeton Language and Intelligence and Bar-Ilan University split the choice into those three questions.

Method

Sequence-level OPD minimizes reverse KL over complete sequences. The practical update is a policy gradient whose reward is the log probability ratio. Token-level OPD matches next-token distributions at each prefix of a student sample. Sampling parameters are stop-gradient, so the prefix probability is only a weight.

The sequence-level gradient adds one term: how much the reverse KL between full continuations falls if this token is chosen. Closer continuations raise the token even when the teacher barely uses it. That is the future agreement bias, the same move as picking a fork because the trail ahead already agrees. Proposition 1 rewrites it as token-level RKL against a teacher tilted by exp(−continuation divergence). Later positions lock in first. A short refusal leaves almost no room for later disagreement, so the bias favors it. A long answer does the opposite.

The divergence changes the landscape. For FKL, the L1 norm of the logit gradient tracks total variation, and the gradient vanishes only as the student approaches the teacher. For RKL it is the fluctuation of the log probability ratio under the student. Mass piled on a few tokens can make that fluctuation arbitrarily small while total variation stays near 1. JS has the same shape, with the teacher replaced by an equal mixture. Per logit, FKL moves with the probability gap. RKL multiplies by the student's own probability, so rare tokens get little force.

Theorem 2 states the speed gap for a tabular student, logits as parameters. Raising a teacher-preferred token from a tiny initial probability p0 to within ε of the teacher takes O(ln(1/p0)) steps for FKL and for an equal FKL+RKL mix, and Ω(p0 to a negative power, divided by ln(1/p0)) steps for RKL and JS. Sequence-level OPD has a matching lower bound. The mix keeps the FKL term. JS does not.

Results

Theorem 1 uses a perfect binary tree of depth L with edges A and B. The target path is all A's. The teacher picks A with probability 1−δ at the root and across the left subtree, δ = exp(−L), and splits evenly after a wrong first step. A tabular student starts uniform. One sequence-level step pushes the wrong first token to at least 1−exp(−Ω(L²)), and success 1−ε takes exp(Ω(L²)) steps. Token-level FKL, RKL, and JS take O(L ln L) and never decrease the correct first token. A Transformer at L = 15 and δ = 10^−4 shows the same order: sequence-level first puts nearly all root probability on the wrong token and repairs it only after later positions match; token-level matches from the root and catches the teacher quickly.

On 6,000 WildJailbreak prompts, half harmful and half benign, a Qwen3-8B teacher refuses 17.5% of benign requests. Sequence-level OPD drives a Qwen3-1.7B student to refuse nearly all benign requests and nearly all harmful ones, so safety accuracy falls. Safety accuracy means refusing harmful requests and answering benign ones. All three token-level divergences raise it. Three seeds agree on the test set. Over-refusal remains for a Gemma-3-4B-it student with a Gemma-3-27B-it teacher.

The bias helps when the easy continuation is correct. If every path that starts with B counts as success, the teacher still assigns only exp(−L) to B, but continuations match there. Theorem 3: sequence-level OPD reaches success 1−ε in one step; token-level FKL and JS need at least Ω(L ln L) steps, RKL at least Ω(ln L). The lead depends on early stopping, and success can fall before it rises. On 6,000 arithmetic problems the teacher calls a calculator on 7.2% of outputs. Sequence-level OPD makes the student call it on nearly every problem and beat the teacher; token-level OPD calls it less and scores lower. Gemma 4B/27B shows the same pattern. Once a call starts, the rest is the expression in the prompt.

New knowledge is code completion: 96 Python APIs released from October 2024 through April 2025, 14 libraries, 480 prompts. The student is Qwen2.5-1.5B-Base from September 2024. The teacher is Qwen3-8B-Base finetuned on those APIs. Where the first token of the correct API name starts at probability at most 10^−3, FKL and the equal mix catch the teacher's naming rate quickly. RKL and JS stay slow. Divergences are computed over the full vocabulary, so the delay is a flat landscape.

On MATH the students already have some skill. Qwen3-0.6B-Base and 1.7B-Base start at 13.0% and 23.3% on MATH500, eight samples per problem. Training uses 7,500 MATH problems and Qwen3 teachers from 4B to 32B. RKL matches or beats FKL at every size, and the gap grows with the teacher. Tokens that rise most under RKL were already plausible to the student. Under FKL they are often initially rare, and FKL accuracy falls as the teacher grows. The main text does not list per-size percentages.

SettingComparisonResult
Binary treeSequence-level vs token-levelSteps to success 1−ε: exp(Ω(L²)) vs O(L ln L)
Benign refusalTeacher at 17.5%Sequence-level refuses nearly all; token-level raises safety accuracy
CalculatorTeacher calls it on 7.2%Sequence-level calls it on nearly every item and beats the teacher
New APIs, first token ≤ 10^−3FKL and equal mix vs RKL, JSThe first two catch the teacher; RKL and JS stay slower
MATH500RKL vs FKL, teachers 4B–32BStart 13.0% / 23.3%; RKL's lead grows with teacher size

Why it matters

Choosing an OPD objective by covering versus mode seeking misses order and step count. Sequence-level and token-level updates can differ by an exponential in the square of the length. FKL and RKL can differ by an exponential in the token's initial probability.

For tokens the student barely uses, such as APIs published after its release, use token-level FKL or an equal FKL+RKL mix. JS is a poor compromise here: its gradient behaves like RKL. When the student already solves about 10 to 20 percent of problems and many solutions are valid, a larger teacher favors RKL, which amplifies solutions the student can already finish. If a short formula is available and unwanted, sequence-level OPD will lean on it. Over-refusal is measured. Excessive abstention on knowledge questions is only flagged. There is still no quantitative selector for a given student, teacher, and task.

Limitations

The exponential step separations are proved for tabular students. The logit-gradient identities hold for any network, but the map from parameters to logits can change which objective wins. The Transformer checks cover tree navigation, refusal, API completion, and MATH only.

Sequence-level gains rely on early stopping. The teacher calls the calculator on 7.2% of outputs, and accuracy after the student is pulled back toward mental arithmetic is not reported. The modified tree has the same dip in success.

A language-model judge labels refusals. Short formulaic refusals are the easiest texts for that judge to spot, and agreement with humans is not reported, so the near-total refusal rate may be inflated. The API teacher was extra-finetuned, while the student is an earlier base model, so missing knowledge and raw capacity move together. Per-size MATH accuracies are not in the main text. Standard sequence-level OPD is reverse KL over full sequences, so the future-agreement bias and the vanishing gradient share one loss. Token-level FKL stays well behaved, so the tree and safety failures cannot be blamed on the divergence alone, but there is no sequence-level run with a different divergence.

Terms

Source

What people are saying

Related papers

All paper explainers