From VPG to GRPO: the clean evolution of RL algorithms behind LLM training
cwolferesearch · x · 2026-09-25
A clear lineage of RL algorithms used to train LLMs: VPG → REINFORCE → PPO → GRPO. VPG gives the policy gradient foundation; REINFORCE makes it computable via Monte Carlo sampling; actor-critic/PPO uses a value function as baseline with clipped conservative updates; GRPO removes the critic entirely, estimating advantages from group-relative rewards over multiple completions per prompt. The progression is driven by reducing gradient variance and training overhead.
Related event: From VPG to GRPO: A Complete Guide to RL for LLMs(2 posts)→
More from Research
- humans& launches Persimmon, a 550B-parameter user model that simulates real human behavior — CShorten30 · 2026-09-25
- SkillRL (NeurIPS 2026): 7B model beats GPT-4o by 41% via recursive skill evolution — cihangxie · 2026-09-25
- NVIDIA's open-source Nemotron-Cascade RL recipe wins NeurIPS Oral, IOI silver — _weiping · 2026-09-25
- Researchers argue latent 'neuralese' reasoning would sharply raise AI misalignment risk — RyanGreenblatt · 2026-09-25
- VeriTile embeds Triton GPU kernels in Lean, with AI agents writing machine-checked correctness proofs — KaiyuYang4 · 2026-09-25
- Jev Does Not Play Dice: 83% confidence, 19% accuracy on a fair die roll — kh-ai · 2026-09-25