Score Centering in Policy Gradients Revealed as a Straight-Through Estimator

A new analysis shows score centering in policy gradients is equivalent to a straight-through estimator, addressing training instability from policy mismatch more fundamentally than importance sampling fixes.

2026-09-20 ~ 2026-09-22 · 3 related posts

• Elegant Derivation Shows Policy Gradient Score Centering Is a Straight-Through Estimator — YouJiacheng · 2026-09-20
• Score Centering is secretly a STE: new fix targets LLM RL training instability — brandondamos · 2026-09-21
• Rethinking Policy Gradients: Score Centering Skips Importance Sampling Entirely — brandondamos · 2026-09-22