Score Centering Stabilizes Off-policy Reinforcement Learning
Martin Marek, Max Ryabinin
cs.LG
2026-09-18
Engine mismatch makes policy gradients distill the trainer toward the sampler. Score centering subtracts expected score; INT8+INT4 30B math RL hits 30% vs 12% for TIS.
LLM reinforcement learning runs two forward passes: a sampler draws rollouts, a trainer computes gradients. When those engines disagree numerically, that is training-inference mismatch (TIM). Sources include quantization, kernels, reduction order, and stale async weights. Killing TIM outright costs rollout throughput. SFT is stable on fully offline data; RL can still reward-collapse. Almost every practical fix is importance sampling plus clip or mask to tame variance, and the clip puts bias back in.
Split the expected policy gradient. The expectation of reward times score equals mean reward times mean score, plus the covariance of reward and score. The first term is drift. Only the covariance is signal.
On-policy, mean score is identically zero because probabilities sum to one. Under TIM, tokens come from sampler q while scores are computed on trainer p, so mean score is generally nonzero. Drift carries no information about which token was better. Its direction is SFT of the trainer toward the sampler, scaled by expected reward at that prefix. The sampler is a biased copy of the trainer and is synced back, so the bias compounds. That is why all-positive rewards are the least stable online setting: every step distills a teacher that tracks the student.
Group centering does not remove drift. Expected advantage at a given prefix is not zero, and drift is largest exactly where the learning signal lives.
Score centering subtracts the expected score under the sampler from each score. The correction is additive, deterministic given the prefix, independent of the sampled token, and has no hyperparameters. The expected update collapses to the covariance term. The only difference from on-policy is that covariance is measured under q, not p. With no TIM, the correction is exactly zero.
The full next-token distribution does not fit in memory: Qwen3's vocabulary is about 152k, so long sequences would need terabytes. The implementation stores the sampler's top-k logprobs (k=128 by default; k=32 matched the full vocabulary in every setting they tried), fills the tail from the trainer scaled to the sampler's tail mass, and writes a scalar loss with stop-gradient. Because the term is additive, it composes with truncated or masked importance sampling (TIS / MIS).
Every method sits on the same objective: REINFORCE with group-centered rewards, swapping only the correction. Qwen3-0.6B trains on Countdown; Qwen3-30B-A3B-Base trains on the INTELLECT-2 math split.
Larger synthetic weight noise makes collapse happen earlier. Under the strongest noise, only score centering and its compositions with TIS or MIS finish training.
Two more realistic TIM sources: quantize the sampler only, or refresh the inference engine every 64 steps. Under quantization, score centering alone is best. Under staleness it wants TIS or MIS on top, because covariance is still measured under q and IS first nudges that distribution toward p.
Three 30B quantization ladders:
| Setting | score centering | TIS | uncorrected PG |
| FP8 sampler | stable, like PG | — | 58% train acc. |
| KV cache to FP4 | 52% | 51% | collapse by step 200 |
| INT8 weights + INT4 KV | 30% | 12% | all others below 5% |
PPO and DAPO survive staleness and die under quantization and weight noise. Their clip was built for ratios from policy movement; numerical error does not produce those ratios.
The collapse story is now one sentence: a teacher that tracks the student, not the absolute size of the mismatch. Offline distillation is stable and online RL is not, which matches the mechanism. Canceling drift with no importance ratios already stabilizes severe quantization, so drift is the cause.
In production this is a few lines of loss and it is orthogonal to IS. For heavy quantized rollouts or long async episodes, start with score centering; stack TIS or MIS when staleness is severe. Matching numerics with batch-invariant kernels, or locking trainer and sampler to one precision, is no longer the only path.
Drift is gone, but the remaining update still measures covariance under the sampler, so severe staleness still needs IS. Headline numbers come from deliberately harsh TIM and short sequences (Countdown max 512 tokens, 30B max 1024) as a proxy for milder TIM over longer training. MoE router indices are not replayed. There is no production run that watches mild TIM accumulate over thousands of steps.