RL training drift equals distillation toward the rollout sampler
An analysis shows RL training drift stems from train-inference mismatch, mathematically equivalent to cross-entropy distillation toward the sampler. Score Centering corrects it by subtracting the drift term, implemented via top-128 vocabulary tokens with rho rescaling.
2026-10-16 ~ 2026-10-16 · 2 related posts
- Score drift in RL fine-tuning is distillation toward a moving teacher, explains researcher — nrehiew_ · 2026-10-16
- How score centering is computed: topk(128) with a rho rescaling trick — nrehiew_ · 2026-10-16