Elegant Derivation Shows Policy Gradient Score Centering Is a Straight-Through Estimator
YouJiacheng · x · 2026-09-20
YouJiacheng shares a compact derivation showing that score centering in policy gradients is equivalent to a straight-through estimator (STE). With s = ∇z log p(a) = ea − p, its expectation under q is s̄ = q − p, so the centered score ea − q equals the STE form zp + sg(zq − zp). The argument applies beyond linear policies; the interpretation was first found by @ACSync, echoing @nanjiangcs's argument.
More from Research
- SGLang Megakernel Pushes Muse Spark to 1,109 tok/s, 2.1x Over Baseline — hsu_byron · 2026-11-12
- ArcGrav: a cross-model Agent Control Language to cut token overhead — ArchitecturalGravity · 2026-11-12
- Open-source dual-arm robot solves tangram puzzles with 90 demos, ~60% success — DanGrover · 2026-11-12
- FutureHouse's deslopper trains Kosmos to write indistinguishably from human experts — proteinrosh · 2026-11-12
- Fleuret amplifies debate: AI can't slash wet-lab experiments in unmapped biology — francoisfleuret · 2026-11-12
- Recursive bets on open-endedness: agents running massive parallel experiments like scientist communities — agihouse_org · 2026-11-12