The Teacher Is a Direction, Not a Destination: Extrapolating RL-Induced Representation Residuals in On-Policy Distillation
Hao Li, MeiJia Chen, Weijie Ren, Donghan Li, Zijun Tian, Jingchun Huang, Naibo Wang
cs.LG
2026-09-29
RIDE extrapolates hidden-state targets along the teacher-minus-pre-RL residual; students beat the RL teacher on all four pairs, where output-space extrapolation degrades.
On-policy distillation (OPD) trains a student on its own rollouts with per-token supervision from a teacher, and it is now a standard post-training stage. Its ceiling is the teacher. Generalized variants (here ExOPD) treat the teacher-to-reference log-probability ratio as an implicit reward and scale it up, which lets a student pass the teacher, but training turns unstable once the coefficient grows.
The paper supplies a mechanistic account. The language-model head is linear, yet it scales directions very unevenly: 79.8% of the teacher-to-base residual's hidden-state energy sits in the head's 512 weakest directions, which contribute only 30.0% of its centered-logit energy. Most of what RL changed inside the model reaches an output-space loss at a small fraction of its weight, and earlier layers are left unconstrained. The sampled-token log-ratio also carries noise, and extrapolation scales that noise by (λ−1)².
RIDE (RL-Induced Direction Extrapolation) works in a specific setting: the teacher is an RL-trained version of a base checkpoint, and the student initializes from that same base, so all three models share a representation space. On each student-generated prefix, the frozen teacher and the frozen base each run one forward pass on identical tokens; the layerwise difference Δ = hT − hB is the RL-induced residual, and the regression target sits beyond the teacher:
h = hT + (λ−1)Δ = λhT + (1−λ)hB
The student's hidden states are regressed toward this target over all layers and the last 2,000 response positions. The choices that matter:
Avg@16 (%) on AIME24, AIME25, and AIMO, mean over three training seeds:
| Method | R1-Distill-1.5B | Qwen3-4B | Llama-3.2-3B | Phi-4-mini |
| RL teacher | 55.30 | 65.59 | 13.01 | 17.96 |
| Untouched student | 39.00 | 62.82 | 6.99 | 15.13 |
| OPD top-1 | 52.90 | 59.24 | 1.08 | 12.90 |
| OPRD | 54.50 | 62.01 | 10.94 | 17.33 |
| ExOPD | 49.87 | 48.75 | 6.95 | 12.22 |
| RIDE | 56.38 | 66.07 | 13.33 | 18.30 |
RIDE is the only method whose mean beats the RL teacher on all four pairs. Its margin over OPRD ranges from 0.97 to 4.06 points, and it leads ExOPD by 9.1 points on average. ExOPD lands below its teacher on every pair, and on Qwen3-4B it sits 14.1 points below the untouched student: when the teacher-to-base gap is small (2.8–6.0 points), sampled noise dominates the extrapolated signal.
The coefficient sweep separates the two spaces cleanly. RIDE rises from 54.3 at λ = 1 to a peak of 55.4 at λ = 1.25, every λ in [1.15, 1.35] beats OPRD, and λ = 2 still finishes at 52.4. ExOPD degrades under every λ > 1: at λ = 2 it falls to 46.2, the format score drops from 92% to 64%, and responses lengthen to 8,700 tokens. Measured conditional variance of the ExOPD advantage grows 12.6× from λ = 1 to λ = 2; RIDE's is zero.
Direction controls rule out a regularization story: a same-norm random vector, a reversed direction, a mismatched-origin checkpoint, and a trajectory-mismatched residual all land within one point of OPRD (54.50); only the true RL residual reaches 56.38.
A minimal, well-attributed change to OPRD: only the target construction differs, and the mechanism analysis plus ablations pin the gain on the RL residual itself. The use cases are concrete, chiefly distilling an expensive RL run back into its base model and merging RL experts that share a base. Within that scope it is currently the only stable way to distill past the teacher. The scope is real, though: teacher and base must share architecture and tokenizer, so black-box or heterogeneous distillation is out.
Author-stated: RIDE needs the pre-RL checkpoint and a shared representation space, uses a single global λ, and was evaluated only on mathematical reasoning with one RL recipe (JustRL). Students are supervised toward hidden states no model ever produced, so calibration and safety are untested, and biases RL introduced in the teacher could be amplified rather than copied. Three further caveats from the numbers: on three of four pairs the margin over the teacher sits within one across-seed standard deviation; the Llama-3.2-3B pair's AIME scores are near the evaluation floor, so that pair's evidence rests on AIMO; and λ = 1.25 was swept on the R1 pair, with no per-pair tuning elsewhere. All conclusions come from a 500-step, 4×H200 budget, and longer-training behavior is unknown.