VASPO Combines Self-Distillation and Privileged Value Estimation to Boost PPO Reasoning
ustc-community · hf · 2026-10-15
Researchers from USTC and iFLYTEK introduce VASPO (Value-Augmented Self-Distillation Policy Optimization), which embeds a self-teacher and critic within PPO to give reasoning models denser supervision. The paper argues on-policy self-distillation signals reflect local token preferences rather than expected return of unguided continuations, and proposes a more effective PPO integration combining token-level advantages with bounded probability ratios, improving accuracy on math benchmarks.
More from Research
- DeepPhylo infers phylogenetic trees without unrealistic evolutionary assumptions — anshulkundaje · 2026-10-15
- Wander preprint explores graph foundation models via shared objective and random-walk interface — mmbronstein · 2026-10-15
- CHI Reviewer: AI Slop Is Real, but Unusually Rigorous Papers Still Getting Through — IanArawjo · 2026-10-15
- Into the Unknown: CMU benchmark finds 114 of 135 AI-written research papers fail integrity checks — arankomatsuzaki · 2026-10-15
- 8 non-negotiable principles for building AI that actually learns from experience — tokenbender · 2026-10-15
- Benchling launches Science Foundries: multiplayer agent swarms for end-to-end science workflows — nlarusstone · 2026-10-15