VPO: A Training Method to Boost Solution Diversity
bronzeagepapi · x · 2026-07-19
Vector Policy Optimization (VPO) is a training method designed to mitigate the issue of "declining solution diversity," a problem that limits the gains achieved during test-time search.
The training pipeline roughly works as follows: the model sequentially generates multiple candidate answers, which are evaluated using multiple objectives and randomized reward weights. The highest-scoring solution for each set of weights is then selected and averaged to produce a scalar reward. The author notes that this reward calculation method can be combined with GRPO.
More from Research
- 'Vibe verification': ~60% of AI-written formal specs prove the wrong property — basedjensen · 2026-10-26
- Ito: a browser that detects browsing intent and auto-organizes tabs into Flows — andrewkuznet · 2026-10-26
- Cornell study says brains process language as linear sequences, challenging Chomsky's hierarchy — anirbanbandyo · 2026-10-26
- Hi3D V1.0 open-sourced with 2M training assets and 90% slicing success rate — kimmonismus · 2026-10-26
- Ortet evaluates 34 open-weight models on 50 health tasks, finds general leaderboards insufficient — elmanmansimov · 2026-10-26
- HackBench injects reward hacks into agent trajectories to stress-test monitors — DanielKhashabi · 2026-10-26