Distilled RL mixes teacher supervision into LLM post-training and beats RL, OPD

ZGCA · hf · 2026-07-21

Distilled RL is a new post-training method for LLMs that tries to combine reinforcement learning with teacher guidance.

The paper argues that plain RL suffers from coarse outcome supervision and hard credit assignment, while on-policy distillation can either teach too little or become ineffective when the teacher is too different.
Distilled RL integrates teacher supervision directly into the RL objective so it can transfer new knowledge more selectively instead of blindly matching teacher logits.
The method includes three components: reverse importance sampling with clipping, negative sample reset, and sequence-level geometric normalization.
In both within-family and cross-family distillation, the authors report substantially better pass@1 and pass@k than standard RL and OPD.

More from Research