SimpleOPD: Simple Tokenizer-Agnostic On-Policy Distillation for Long-Context Reasoning
Haonan He, Haodi Lei, Yun Luo, Haoran Zhang, Shunkai Zhang, Yizhuo Li, Shengji Tang, Zhilin Wang, Runzhe Zhan, Lei Bai, Ganqu Cui, Fangchen Yu, Yafu Li, Peng Ye, Ning Ding, Yu Cheng
cs.CL, cs.AI
2026-08-14
SimpleOPD aligns teacher and student tokens by matching identical text spans, lifting Intern-S2-Preview's ProofBench score from 21.7 to 44.5, closing most of the gap to a 158B teacher.
On-policy distillation works well for transferring reasoning from a strong teacher to a weaker student, but it normally requires both to share a tokenizer so token-level probability distributions can be compared directly. The strongest reasoning models rarely match the tokenizer of whatever student a team actually wants to deploy, and long-context teachers paired with short-context students make things worse. This paper tackles a specific hard case: transferring proof-reasoning ability from a 30B long-context math model, SU-01, to short-context students from different model families (Qwen3.5, Intern-S2, GLM-4.7, Gemma-4), each with its own tokenizer. Naive distillation in this setting breaks down: student output length balloons, truncation gets frequent, and training destabilizes, largely because the student stops reliably emitting termination tokens like </think> and <|imend|>.
The core move is aligning at the text level instead of the token level. The student generates a response with its own tokenizer; the resulting text string is then re-encoded under the teacher's own tokenizer and chat template, giving the teacher its native token sequence over the same text. The two token sequences won't line up one-to-one, but a two-pointer scan finds positions where a student token and a teacher token cover exactly the same text span. Those get treated as aligned, and the teacher's log-probability at that position becomes the supervision signal. Unmatched positions fall back to the student's own probability, contributing no extra signal. This sidesteps the shared-vocabulary requirement entirely, relying only on the weak assumption that both sides produce the same underlying text.
That alone isn't enough. Naive training still causes output length to balloon and termination to degrade. Two stabilization techniques fix this: masking the advantage on structural termination tokens (</think>, <|imend|>) so the student isn't forced to fully mimic the teacher's termination behavior, and adding a KL-divergence loss against the student's own reference policy to keep training inside a trust region. Together, truncation rate drops to near zero.
| Model | ProofBench@4 | AnswerBench@8 | AIME25@8 |
| SU-01 (teacher, 30B) | 45.00 | 77.50 | 94.60 |
| Intern-S2-Preview (base) | 21.70 | 76.03 | 88.33 |
| Intern-S2-OPD (after SimpleOPD) | 44.50 (+22.80) | 80.10 (+4.07) | 95.00 (+6.67) |
Intern-S2-OPD nearly matches the teacher SU-01 on ProofBench (44.50 vs 45.00) and surpasses it on AnswerBench and AIME25. Under an independent Gemini-2.5-Pro judge, Intern-S2-OPD improves from 34.0 to 55.2, beating GPT-5 and Gemini-2.5-Pro, trailing only SU-01 and DeepSeek-V3.2-Speciale. The method also works within the same tokenizer family: Qwen3-4B-OPD gains 12.30 points on ProofBench. Ablations show both stabilization techniques matter: termination-token masking alone still lets length balloon; only adding the reference KL pushes truncation to near zero. Cross-family results (GLM-4.7, Gemma-4) show transfer quality tracks tokenizer similarity: GLM (also BPE-based) improves on both benchmarks, while Gemma (SentencePiece, structurally different vocabulary) actually loses 1.3 points on AnswerBench. Training data was exclusively math proofs, yet the gains generalized to out-of-domain science benchmarks like HLE and HiPhO (HiPhO rising from 38.6 to 41.1), suggesting the distillation transfers something beyond math-specific tricks.
For teams wanting to distill a strong reasoning model into a short-context deployment model from a different family, this offers a practical recipe for token-level supervision without a shared vocabulary, plus two easy-to-miss stability traps: the student's termination behavior gets dragged off by the teacher's long outputs, and unregularized policy updates drift and destabilize. Both failure modes look, on a bare score curve, like "it's working": the paper's case studies include a student that got stuck in 972 repeated self-checks and another that collapsed into a one-token loop, the kind of degeneration a score-only view would miss entirely.
The cross-family evaluation covers only two models (GLM-4.7, Gemma-4), and already shows gains shrink or turn negative as tokenizer dissimilarity grows — whether the method holds up under more extreme tokenizer mismatches (non-BPE schemes, for instance) is untested. Every experiment distills from the same teacher, SU-01, and the transferred capability is math proof reasoning specifically; the paper notes longer distillation sequences matter more for complex proofs but doesn't test whether other long-horizon reasoning domains (agentic tasks, say) need similar adjustments. The text-level alignment also depends on both sides decoding to an identical surface string — if teacher and student normalize text differently (whitespace, Unicode), alignment would silently degrade to unsupervised positions, and the paper doesn't quantify how often this happens.