TPPO Tracks Value Evolution Better Than PPO, Gains Up to 8.4 Points on Math Reasoning
Futing Wang · hf · 2026-10-14
TPPO (Tracking-Value PPO) addresses the failure of standard critics to track value evolution along trajectories under sparse terminal rewards. The authors show successive prefix values admit a Bayesian decomposition with additive updates in log-odds space, and use this structure to build a cumulative value parameterization that sums position-wise scalar outputs into value logits, learned via terminal-outcome supervision.
TPPO improves aggregate math reasoning scores over PPO by 7.0 and 8.4 points on Qwen3-4B-Base and Qwen3-8B-Base respectively, with better out-of-distribution generalization. Gains extend to Qwen3-4B, Qwen2.5-Math-1.5B, Qwen3.5-9B, and to countdown and scientific coding tasks. Analysis shows more accurate value tracking and more reliable advantage estimates. Code is public.
More from Research
- Tsinghua professor's ApexIntelligence raises $50M to build self-evolving foundation models — DeryaTR_ · 2026-10-16
- Anders Sandberg on fluid computation: error budgets tighten quadratically over time — anderssandberg · 2026-10-16
- How the fluid computation works: force field advects particle to 0/1 boxes — anderssandberg · 2026-10-16
- FlashAttention creator Tri Dao joins Radical Numerics as scientific advisor — exnx · 2026-10-16
- SAIR Launches Open Math OS, Offering Free Compute and 100+ Tools for Community-Owned Math AI — Thom_Wolf · 2026-10-16
- JapanFold Claims One Tenstorrent Server Beats an Nvidia DGX B200 on Bio Models at a Third of the Price — DavidBennett__ · 2026-10-16