Score centering, rumored OpenAI RL trick, tolerates staleness and may rescue slow chips
teortaxesTex · x · 2026-11-21
Quoting a ByteDance engineer's thread on two new RL algorithms—RIPO (Riemannian Isometric Policy Optimization, validated at scale for agentic RL) and score centering (rumored to be OpenAI's ultimate RL algorithm)—teortaxesTex notes score centering is now widely adopted, e.g. StableReinforce in Tinker. The attached paper shows that under severe quantization (INT8 sampler) and staleness (inference updated only every 64 steps), score centering composed with TIS/MIS performs best while PPO/DAPO collapse under quantization noise. The authors recommend composing it with importance-sampling correction when the trainer drifts far from the sampler. It can tolerate tens of updates of median staleness and might rescue slower chips.
More from Infra
- Inkling 2 Mini quantized to 5.4GB/8.9GB, keeping 91-97% performance — teortaxesTex · 2026-11-21
- Open-weight models are less efficient, so their rise may increase chip sales — aronchick · 2026-11-21
- $31/month quad-channel EPYC + 128GB runs 3-bit Qwen at 128k context — Ne00n · 2026-11-21
- Ternary Bonsai-2 vs Qwen3.5-2B on A13/A14/M3: 32-55% faster decode, and why thermal logging matters — leo-k7v · 2026-11-21
- Essay 'The Organic Web' weaves 62 citations to dissect data's physical weight — round · 2026-11-21
- ARK projects Starship marginal launch cost falling to $358/kg by 2027 — mitchdeg · 2026-11-21