Paper quantifies on-policy distillation's tolerance to training-inference mismatch

Cong Yu · hf · 2026-11-10

Question: on-policy distillation (OPD) trains a student LLM on its own responses scored by a fixed teacher; systems cut cost via quantized inference and asynchronous rollouts, but both introduce training-inference mismatch that can destabilize RL. How much can OPD tolerate?
Method: all mismatch sources measured on one scale — quantization across three student-teacher pairs, plus weight staleness and injected noise on one pair.
Findings: OPD tolerates substantial mismatch without meaningful quality loss, degrading only past an empirical mismatch budget; cost and degradation depend on the model, with some layers especially sensitive to reduced precision; OPD stays stable under rollout configs that destabilize standard RL methods.
Practice: a lightweight admission check tells you if a given inference config is within tolerance, and FP8 recipes exploit the tolerance for cheaper training.
Authors from Aalto, NTU, Cambridge, BIT, et al. (arXiv, Nov 8, 2025).

More from Infra