RIDE: Extrapolating RL-Induced Representation Residuals Lets Distilled Students Match or Beat RL Teachers

Hao Li · hf · 2026-10-01

Problem: Output-space extrapolation in on-policy distillation is unstable because the LM head attenuates hidden-state changes anisotropically and log-prob ratios inject amplified noise.
Key observation: RL shifts a model's internal representations relative to its base checkpoint, and this shift direction is measurable at every layer.
Method (RIDE): Extrapolate the RL-induced change directly in representation space — at each layer and token position, compute the residual between the teacher and its pre-RL checkpoint, then regress the student's hidden states toward targets displaced beyond the teacher along that residual. This is equivalent to maximizing a linear directional reward under a quadratic penalty centered at the teacher.
Results: Across four base/RL-teacher pairs spanning scales, architectures, and pre-training lineages, RIDE approaches or exceeds the RL teacher on every pair (the only method to do so on average) and consistently outperforms output-space extrapolation, which degrades students when the teacher is near its base.

More from Research