Score Centering fixes off-policy baseline drift in RLHF, full derivation

青稞AI · wechat · 2026-09-26

A deep derivation-heavy walkthrough of the paper "Score Centering Stabilizes Off-policy Reinforcement Learning," plus a full formalization of combining it with TIS/MIS for RLHF and agent training.

Core problem

On-policy, any state-dependent function is a valid baseline. But once samples come from a behavior policy while scores come from the target policy, its conditional expectation is no longer zero — swapping baselines genuinely changes the gradient, not just the variance
The naive off-policy update implicitly contains a reverse-KL gradient term; shifting reward/value zero-points can even flip the update direction (mean-score drift)

Key results

Score Centering (subtracting the score's expectation under the sampler) restores baseline invariance and is expectation-equivalent to an exact prefix-level value baseline without training a critic
Plain advantage updates and score-centering updates are generally not the same
Exact importance sampling already restores zero mean, so the centering correction vanishes; TIS/MIS leave residual drift from clipping/masking and do benefit from centering

Engineering notes

The paper stores only top-k sampler log-probs and reconstructs the tail with the trainer distribution, so corrections only need the top-token graph
All coefficient quantities must be stop-gradient, including correction terms
Diagnostic: constant or shuffled reward should yield near-zero average updates
Limitation: token-level IS fixes only the current action distribution, not future continuation distributions

Original post →

Score Centering fixes off-policy baseline drift in RLHF, full derivation

More from coding & agent