VPO: A Training Method to Boost Solution Diversity
bronzeagepapi · x · 2026-07-19
Vector Policy Optimization (VPO) is a training method designed to mitigate the issue of "declining solution diversity," a problem that limits the gains achieved during test-time search.
The training pipeline roughly works as follows: the model sequentially generates multiple candidate answers, which are evaluated using multiple objectives and randomized reward weights. The highest-scoring solution for each set of weights is then selected and averaged to produce a scalar reward. The author notes that this reward calculation method can be combined with GRPO.
More from Research
- KAN author Ziming Liu warns AutoResearch generates chaos, not insight — ZimingLiu11 · 2026-10-27
- Diamandis: AI Cancer Navigator and $100 Genome Sequencing Close the Care Gap — PeterDiamandis · 2026-10-27
- EMNLP Oral: training-free Nexus Sampling cuts KV-cache memory 10x; OpenResearcher tops BrowseComp-Plus at 54.8% — yuz9yuz · 2026-10-27
- ICML 2026 position paper: LLM 'thinking traces' are a harmful anthropomorphism — rao2z · 2026-10-27
- Forking gap follows a power law in context frequency; masking rare n-grams shrinks it — ZimingLiu11 · 2026-10-27
- How forking happens: n-gram tables split logits into non-interacting subspaces — ZimingLiu11 · 2026-10-27