STOPD cuts on-policy distillation rollout tokens by 25–48% with unbiased random truncation, 2.2x speedup

机器之心 · wechat · 2026-11-19

Researchers from Cornell and Georgia Tech propose STOPD to slash the cost of on-policy distillation (OPD), where each training round requires full student rollouts plus per-token teacher feedback.

Method: sample a stopping position before generation, truncate rollouts early, and compensate sampling bias via inverse-probability weighting; block truncation (default width 128) caps tail weights from 16,384 down to 128. They prove the weighted gradient sum matches full rollouts in expectation, with per-step error bounds for SGD, Adam, and Muon.

Results (Qwen3 teachers, 1.7B–8B students): math speeds up 1.8x with 34% fewer tokens, code 2.2x with 48% fewer, instruction-following 1.3x with 25% fewer — accuracy on par with vanilla OPD throughout. Larger batches damp loss jitter (12.2%→6.0% at batch 512). Unlike fixed-length truncation (ESR), STOPD preserves supervision on late tokens including natural EOS, keeping stopping behavior healthy.

Paper and code are open-sourced; the authors see applications to other token-level supervised training like SFT and offline distillation.

Original post →

More from Research

Research channel →