Off-policy drift emerges as the central problem in post-GRPO LLM RL research
Since GRPO's rise, the mismatch between training and inference policies has become the central issue in LLM RL research: the Score Centering paper proposes an additive correction to stabilize off-policy training, while a 48-page long-read surveys 14 correction methods.
2026-10-31 ~ 2026-10-31 · 3 related posts
- Episode 1: Tinker's StableReinforce Reverse-Engineered as REINFORCE with Score Centering(2026-10-25, 2 posts)
- Episode 2: TRL Makes Score Centering + Masked Importance Sampling Default for Async GRPO(2026-10-27, 5 posts)
- Episode 3: Off-policy drift emerges as the central problem in post-GRPO LLM RL research(2026-10-31, 3 posts)
- natolambert: importance-sampling fixes for train-inference mismatch dominate RL research — natolambert · 2026-10-31
- Score Centering paper: additive correction stabilizes off-policy RL under train-inference mismatch — natolambert · 2026-10-31
- 48-page deep dive: why modern LLM RL is secretly off-policy and 14 ways to fix it — cwolferesearch · 2026-10-31