willcb: RL Is Direct Objective Optimization, On-policy GRPO Comes Closest

willcb argues that RL is direct estimation of the objective gradient and other algorithms merely evade direct optimization via proxy objectives; he considers on-policy GRPO without std normalization very close to "true RL" (REINFORCE), with variants like DAPO being efficiency tricks.

2026-10-12 ~ 2026-10-12 · 2 related posts

• willcb: every non-RL algorithm is a proxy — policy gradients are the minimum-variance unbiased estimator — willcb · 2026-10-12
• willcb: on-policy GRPO is nearly true RL, everything else is efficiency tricks — willcb · 2026-10-12