Score centering, rumored OpenAI RL trick, tolerates staleness and may rescue slow chips

teortaxesTex · x · 2026-11-21

Quoting a ByteDance engineer's thread on two new RL algorithms—RIPO (Riemannian Isometric Policy Optimization, validated at scale for agentic RL) and score centering (rumored to be OpenAI's ultimate RL algorithm)—teortaxesTex notes score centering is now widely adopted, e.g. StableReinforce in Tinker. The attached paper shows that under severe quantization (INT8 sampler) and staleness (inference updated only every 64 steps), score centering composed with TIS/MIS performs best while PPO/DAPO collapse under quantization noise. The authors recommend composing it with importance-sampling correction when the trainer drifts far from the sampler. It can tolerate tens of updates of median staleness and might rescue slower chips.

Original post →

More from Infra

Infra channel →