MIT's Vector Policy Optimization replaces scalar RL rewards to boost LLM test-time search, accepted at NeurIPS 2026

kastnerkyle · x · 2026-11-07

Vector Policy Optimization (VPO), a paper from MIT's Improbable AI Lab and collaborators, has been accepted to NeurIPS 2026; the authors released code and a project website.

Core claim: modern LLM pipelines rely on test-time search (best-of-n, verifier reranking, AlphaEvolve-style evolutionary search), which only pays off when candidates are diverse. But conventional RL like GRPO collapses all reward signals upfront into a single scalar, pushing the model toward one high-probability answer — so sampled candidates are near-duplicates and search has nothing to choose between.

Method & results:

VPO keeps rewards as vectors, simultaneously optimizing across different reward weightings so solutions spread across the frontier of good trade-offs;
It is a drop-in replacement for the GRPO advantage estimator;
It improves pass@k on LiveCodeBench, with the gap growing as search budget grows;
Inside an AlphaEvolve-style search loop, VPO cracks hard problems GRPO never solves.

Original post →

MIT's Vector Policy Optimization replaces scalar RL rewards to boost LLM test-time search, accepted at NeurIPS 2026

More from Models