Score Centering tackles the root cause of RL instability when train and sample policies differ
_AndrewZhao · x · 2026-09-19
- LLM RL is notoriously unstable when the training and sampling policies use different numerics; standard fixes (matching numerics, importance sampling) only work around the problem.
- The thread derives the root cause from first principles and proposes directly cancelling it.
- Score Centering is simple to implement, competitive on its own, and compatible with existing approaches.
More from Research
- Anthropic says it may imminently hit RSP threshold for AI-accelerated R&D, releases provable inference work — tobyordoxford · 2026-11-04
- Google scanned 1,033 real household products, packed into a 345 MB GLB for three.js — victormustar · 2026-11-04
- These 10 GPU optimization papers are why your model runs fast today — NandoDF · 2026-11-04
- INT8 isn't the problem: I benchmarked samplers on INT8 flow models — YakovSava · 2026-11-04
- Karpathy Boosts Open Hacker News Dataset: 10M+ Posts Usable as a Personal Knowledge Base — huangyun_122 · 2026-11-04
- PixelCraft: A Multi-Agent Framework Targets Multimodal AI's Chart-Reading Failures — burkov · 2026-11-04