Rethinking Policy Gradients: Score Centering Skips Importance Sampling Entirely
brandondamos · x · 2026-09-22
Yifei Wang highlights a discussion by YouJiacheng on rethinking policy gradients: instead of optimizing Ep[R] with high-variance importance sampling, score centering optimizes Eq[R] — samples already come from q, so no IS is needed. Since q (inference engine/old policy) can't be optimized directly, p's gradient serves as a biased surrogate, which is exactly a STE. YouJiacheng shows this extends beyond linear policies: for gradients w.r.t. logits, the centered score equals ea - q, making score centering the estimator zp + sg(zq - zp). A perspective shift from optimizing the p-objective to the q-objective that yields major simplification.
More from Research
- AlphaGenome2 hits SOTA linking distal enhancers to target genes from gradients — anshulkundaje · 2026-11-12
- AlphaGenome2 ablations: early encoder conditioning nearly doubles eQTL recall — anshulkundaje · 2026-11-12
- Author says his CHI2027 ADR paper was better than papers he reviewed — windx0303 · 2026-11-12
- Google launches AlphaGenome 2 to predict how DNA variants affect specific cell types — sundarpichai · 2026-11-12
- Stanford's James Zou joins The Economist podcast on AI agents in science — anshulkundaje · 2026-11-12
- New learning rule boosts Hopfield network memory capacity over 10×, per Nature Communications study — bravo_abad · 2026-11-12