Score Centering is secretly a STE: new fix targets LLM RL training instability
brandondamos · x · 2026-09-21
@mryabinin traces LLM RL instability — when training and sampling policies differ — to first principles, proposing Score Centering to directly cancel it, simpler and compatible with importance sampling. @YouJiacheng notes the trick is secretly a straight-through estimator: zp + sg(zq - zp).
More from Research
- SGLang Megakernel Pushes Muse Spark to 1,109 tok/s, 2.1x Over Baseline — hsu_byron · 2026-11-12
- ArcGrav: a cross-model Agent Control Language to cut token overhead — ArchitecturalGravity · 2026-11-12
- Open-source dual-arm robot solves tangram puzzles with 90 demos, ~60% success — DanGrover · 2026-11-12
- FutureHouse's deslopper trains Kosmos to write indistinguishably from human experts — proteinrosh · 2026-11-12
- Fleuret amplifies debate: AI can't slash wet-lab experiments in unmapped biology — francoisfleuret · 2026-11-12
- Recursive bets on open-endedness: agents running massive parallel experiments like scientist communities — agihouse_org · 2026-11-12