Rethinking Policy Gradients: Score Centering Skips Importance Sampling Entirely
brandondamos · x · 2026-09-22
Yifei Wang highlights a discussion by YouJiacheng on rethinking policy gradients: instead of optimizing Ep[R] with high-variance importance sampling, score centering optimizes Eq[R] — samples already come from q, so no IS is needed. Since q (inference engine/old policy) can't be optimized directly, p's gradient serves as a biased surrogate, which is exactly a STE. YouJiacheng shows this extends beyond linear policies: for gradients w.r.t. logits, the centered score equals ea - q, making score centering the estimator zp + sg(zq - zp). A perspective shift from optimizing the p-objective to the q-objective that yields major simplification.
More from Research
- Combining verifiable rewards with rubrics for more efficient model grading — stochasticchasm · 2026-09-22
- Math Isn't Solved: AI Turns a Universe We Walked Into Wormholes — ignite_intelligence · 2026-09-22
- JevBench v1.3.0 Released: Original Jev Still Leads at 74.4, 47 Rivals Closing In — airesearch12 · 2026-09-22
- PufferLib author: 420-step, 4k-rollout config trains Breakout in 1 second on 1 GPU — jsuarez · 2026-09-22
- DeepSeek vs Jev: How an LLM Stacks Up on a System-One Probability Benchmark — frappuccinoCoin · 2026-09-22
- Mitigating reward hacking: classifying frontend design as visual agent tasks with groupwise grading — stochasticchasm · 2026-09-22