Beyond GRPO: MIT's Vector Policy Optimization keeps solution diversity in RL post-training
yacinelearning · x · 2026-10-16
Yacine breaks down Vector Policy Optimization (VPO) from Ryan Bahlous-Boldi and collaborators at MIT, with a 2h04m deep-dive podcast (full chaptered timeline included).
The problem: GRPO-style post-training tends to collapse the model onto a tight set of answers ("spooky sharpening" of the distribution). That's fine when you want convergence, but models increasingly run in long-horizon harnesses that require composing diverse solutions to stumble on discoveries — a distribution too cooked to produce varied solutions is suboptimal.
How VPO works:
- Keep rewards vector-valued instead of collapsing them to scalars
- Leverage in-context learning to hold multiple draft solutions in context
- Sweep all possible weightings across reward dimensions to build a vector maximizing every dimension; take the mean as the post-training objective
The result covers the frontier of solutions and cracks hard problems GRPO can't. The conversation also covers test-time search requiring diversity training, hindsight replay and reward shaping, harness-level fine-tuning, and ablations.
Related event: MIT's VPO: Post-Training Method to Preserve Output Diversity in LLMs(4 posts)→
More from Research
- Open-source 12-chapter guide builds a modern LLM from scratch with every line commented — tom_doerr · 2026-10-16
- John Nash's 1998 Hierarchical Introspective Logics paper recirculates in AI circles — fkasummer · 2026-10-16
- Ex-OpenAI interp lead: most neurons in trillion-param models are now legible — CatAstro_Piyush · 2026-10-16
- AI safety researcher: most technical safety topics echo classic statistics from 20-70 years ago — _onionesque · 2026-10-16
- JevAgentHarness: a 4B decision model lets cheap agents offload tool choices from the LLM — youjiaxuan · 2026-10-16
- Google-led paper uses sequential auctions to organize decentralized LLM agent workflows — _onionesque · 2026-10-16