Score drift in RL fine-tuning is distillation toward a moving teacher, explains researcher
nrehiew_ · x · 2026-10-16
nrehiew breaks down the mathematical root of training-inference mismatch (TIM) drift in RL fine-tuning: the drift term (the score under the sampler) is effectively cross-entropy distillation toward the sampler. Because the rollout model acts as a moving teacher, this distillation destabilizes training.
He notes the drift can no longer be canceled with the log-derivative trick under TIM, since the reward is sampled under the sampler while the score expectation is under the trainer. Part of a thread analyzing Together AI's score centering paper.
Related event: RL training drift equals distillation toward the rollout sampler(2 posts)→
More from Research
- 200 chess games suggest in-context learning doesn't work yet, experimenter says — akbirthko · 2026-10-16
- Mosaic: multi-session agentic RL with hierarchical advantage estimation — LARK-Lab · 2026-10-16
- Meituan LongCat's WorldCast benchmark: GPT-Image-2.5 tops at 72.8, open-source lags — meituan-longcat · 2026-10-16
- Recursive Environment Scaling tanks six models' success rates by 57–77 points — Lingxiao Jing · 2026-10-16
- Harvard's MPT: decentralized agent swarms learn self-orchestration via bidding — Harvard · 2026-10-16
- MIT Biology and Schwarzman College hiring tenure-track professor in computational biology with generative AI focus — anshulkundaje · 2026-10-16