Rethinking Cross-Tokenizer On-Policy Distillation: From Alignment Coverage to Supervision Reliability
Bingxi Hou, Guochao Jiang, Guofeng Quan, Weiqing Li, Wenfeng Feng, Guohua Liu, Yuewei Zhang
cs.CL, cs.AI
2026-10-06
On three pairs, strict 1:1 covers 85.57-96.98% of student tokens. Top-16 keeps at least 96% of the full OPD gain; span MSE lowers accuracy in all 18 settings.
On-policy distillation (OPD) samples answers from the student and applies the teacher's next-token distribution only on prefixes the student actually visits. A shared tokenizer makes reverse KL straightforward. A different tokenizer does not: the same string is cut at different boundaries, and the two predictive distributions sit on different vocabularies.
ULD matches sorted probabilities and ignores token identity. SimCT packs misaligned spans into multi-token units. Byte-prefix marginalization spreads teacher probability onto student tokens. The open question is whether a fuller alignment produces a better student.
Alignment coverage counts how much of the string and the vocabularies can be compared. Supervision reliability asks whether the teacher target helps at the states the student visits. Jaccard overlap weights every vocabulary entry equally. On real student trajectories, frequency and probability are highly uneven. A large vocabulary gap can coexist with strict alignment at most positions.
Three teacher-to-student pairs are used: Qwen2.5-7B-Instruct to Llama-3.2-3B-Instruct, Granite-4.1-8B to Phi-4-mini-instruct, and Granite-4.1-8B to Qwen2.5-7B-Base. The last student is a base model. The first two are instruction-tuned.
The student samples a response. Both tokenizers resegment the decoded text, and only shared boundaries are kept. A strict 1:1 group is one student token and one teacher token over the same span. A mismatch group needs more than one token on at least one side.
The strict loss is reverse KL, KL(student || teacher), and it is applied only at 1:1 positions. Shared entries are matched by token string, after dropping special tokens and ambiguous matches. Both distributions are clipped to that intersection and renormalized. Same-tokenizer OPD uses this same divergence; the support is now the intersection. Reverse KL penalizes student mass where the teacher puts almost none.
Mismatch groups have no token-wise pairing, so the extra loss compares the whole observed path. Token probabilities multiply into a span probability, and student and teacher log-probabilities are fit with mean squared error. The total loss is the strict KL plus λ times that span term. λ = 0 uses only strict positions. Any positive λ includes every mismatch group, which makes structural coverage 100%. λ runs from 0 to 1.5 in steps of 0.25.
A second family keeps the strict positions and shrinks the support. At each position the student's own probabilities select the top-k shared entries, k = 16 or 128. Both sides are renormalized on those k entries before reverse KL.
The pool is 20,000 prompts: 10,000 math prompts from DAPO-Math-17K and 10,000 code prompts from CodeForces. Training runs for 100 on-policy steps at batch 512, learning rate 1e-6, temperature 1.0, top-p 0.95, and a generation cap of 8,192 tokens. Math is seven benchmarks at 32 samples per problem (MATH500, GSM8K, AIME 2024 through 2026, AMC23, Minerva-Math). Code is three benchmarks at 8 samples (HumanEval, MBPP, LiveCodeBench). Full weights the two domain means equally.
Across a full training run, vocabulary Jaccard is only 39.49-64.87%. Strict coverage is still 85.57-96.98% of student tokens and 82.82-97.26% of teacher tokens.
| Pair | Jaccard | Student strict | Teacher strict |
| Qwen to Llama | 64.32% | 93.56% | 85.91% |
| Granite to Phi | 39.49% | 96.98% | 97.26% |
| Granite to Qwen | 64.87% | 85.57% | 82.82% |
Granite to Phi has the smallest overlap and the highest coverage on both sides. Across three training windows, student coverage moves by at most 1.43 percentage points.
Before distillation, one rollout per prompt is scored at strict positions. The shared vocabulary holds 99.69-99.90% of teacher probability mass and 98.99-99.81% of student mass. A student-chosen top-16 keeps at least 93.54% of teacher mass and 94.55% of student mass in every pair. Expanding from 16 to 128 adds at most 3.44 points of teacher mass and 2.70 of student mass.
| Method | Q to L | G to P | G to Q |
| Base | 26.96 | 36.55 | 29.56 |
| ULD | 29.12 | 38.63 | 27.12 |
| Extended ULD | 31.13 | 38.35 | 32.57 |
| GOLD | 27.16 | 41.75 | 46.55 |
| SimCT | 31.59 | 41.68 | 46.10 |
| Strict full | 32.86 | 42.49 | 47.35 |
| Strict top-16 | 32.64 | 42.26 | 47.06 |
| Strict top-128 | 32.38 | 42.54 | 47.65 |
These are Full scores. Strict full gains 5.90, 5.94, and 17.79 points over the undistilled students. Top-16 keeps at least 96% of those gains. On every pair, all three strict variants beat ULD, Extended ULD, GOLD, and SimCT on both the math average and Full. Top-16 leads the strongest baseline by 0.51-1.05 Full points. Code is less tidy: on Qwen to Llama, Extended ULD scores 39.26 on code, against 39.11 for strict full.
Mid-difficulty sets carry most of the visible math gain. On Qwen to Llama, GSM8K moves from 63.44 to 80.21 (teacher 89.93), MATH500 from 34.32 to 52.74 (teacher 74.50), and AIME-2024 only from 2.29 to 4.69 (teacher 11.88). MBPP falls from 53.27 to 51.46. On Granite to Qwen the student starts as a base model, and code rises from 34.51 to 54.84 (teacher 63.32).
Span MSE completes coverage and lowers accuracy. Math, code, and Full all peak at λ = 0 on every pair. The 18 positive weights sit 0.27-1.20 points under their strict baselines. At λ = 1.5, Full drops by 1.02, 0.80, and 0.80 points.
Gradients come from checkpoints trained with the strict loss alone. The cosine between span gradients and strict gradients stays near zero, and Granite to Qwen starts at -0.360. A split of the same strict positions into two random halves, used as a reference, has the higher cosine at every measured checkpoint. From step 0 to step 100, in the same pair order as above, the span-to-strict norm ratio goes 0.294 to 1.942, 0.008 to 0.233, and 0.015 to 0.269.
A larger teacher repeats the ranking. Qwen3-235B-A22B-Instruct-2507 teaching Granite-4.1-8B gives strict top-16 a Full score of 62.46, against base 51.31, SimCT 56.25, and the teacher 74.17. That math mean adds two HMMT 2025 contests. On ALFWorld, a text household environment, Qwen3-4B-Instruct-2507 to Llama-3.2-3B-Instruct, strict top-16 reaches 30.34% overall success, SimCT 28.52%, and the undistilled student 10.68%. Unseen success is 34.64%, above the teacher's 29.43%. Seen success is 26.04%, below the teacher's 29.69%. ULD and GOLD fall to 1.43% and 2.61%.
Splitting responses on shared boundaries and running reverse KL on the shared vocabulary, only at 1:1 positions, already beats the four cross-tokenizer baselines implemented here on Full. Cutting each position to a student-chosen 16 entries keeps at least 96% of that Full gain.
All 18 positive values of λ lose to strict-only supervision, by 0.27-1.20 points. SimCT's own paper reports that recovering excluded supervision can beat shared-vocabulary OPD. Here SimCT beats the undistilled student and loses to strict reverse KL. Hitting 100% coverage does not decide whether the extra loss belongs in the objective.
Top-16 does not remove the teacher forward pass. The student logits still have to be computed before those 16 entries can be chosen.
The gradient comparison uses strict-only checkpoints, and the responses are held fixed while differentiating. It is not measured inside joint training at λ > 0. The pattern lines up with the accuracy drop. Causation is not nailed down. The paper says the diagnostics may help explain the drop.
The drop is tied to mean squared error on span log-probabilities. Masking whitespace, or matching distributions over multi-token units, is not swept. The result does not show that every mismatch objective is harmful.
Probability mass is measured before distillation. Coverage stays stable through training. Whether a top-16 still holds more than 90% of the mass at later steps is not measured again.
The main tasks are verifiable math and code. ALFWorld adds a single pair. Unseen success beats the teacher and seen success does not, over 128 episodes averaged across three evaluations, so that margin is noisy. There is no general chat, multilingual, or long-context evaluation. The 235B teacher run reports only strict top-16, with no λ sweep and no gradients. The main students are 3B to 8B.
Differences of a few tenths of a point have no error bars. Agreement across 18 settings carries more weight than one head-to-head. 32.64 versus 32.86 should not be ranked.
The scorer moves the absolute numbers. Math answers longer than 300 characters are truncated before Math-Verify. HumanEval and MBPP use the original tests, not the EvalPlus augmented sets. Prompts are not deduplicated and are filtered at 2,048 student tokens, so the three students do not see the same set. ULD, Extended ULD, and GOLD drop the terminal EOS from the distillation loss. The strict objective keeps paired EOS positions.
The base student, Qwen2.5-7B-Base, gains 17.79 Full points. The two instruction-tuned students gain 5.90 and 5.94. How much of the base-model jump is reasoning, and how much is answer format, is not separated. On Qwen to Llama the aggregate rises, MBPP stays below the undistilled baseline, and AIME remains far from the teacher.