UIUC's Proximal KL replaces PPO clipping in off-policy RLVR, beats GRPO by up to 2.58%

UIUC-CS · hf · 2026-10-15

Key points

• UIUC researchers (Hao Bai, Jiarui Yao, Tong Zhang) study token-wise importance sampling in off-policy RLVR, which avoids multiplying likelihood ratios across a response but generally biases policy gradients.
• Theory: under coverage and local expressiveness conditions, adding full future KL credit makes the exact proximal optimizer the unique stationary policy, even with mixtures of older policies—no full-response weights or fitted critic needed. They also prove convergence of continuous-time tabular updates and finite-sample reward guarantees for linear logistic models.
• Proximal KL in practice separates the proximal reference from the sampling policy and replaces PPO clipping with bounded future KL credit.
• Across seven matched off-policy comparisons on Qwen3-1.7B/4B, Proximal KL beats GRPO on eight math benchmarks by up to 2.58% micro-averaged accuracy, avoids collapses seen in PG and CISPO, and is less sensitive to baseline perturbations.

Original post →

More from Research

• NCM: new text generation paradigm up to 31x faster than diffusion LMs with better quality — simonguozirui · 2026-10-15
• AlphaNIST: one RL policy solves all 20 NIST assembly tasks with 98.2% success in simulation — hbouammar · 2026-10-15
• Neural field planning for hybrid manufacturing cuts estimated build time 68.3% — jiqizhixin · 2026-10-15
• Tev-Gym: 1,170 dense-reward RL environments push long-horizon agentic training — RulinShao · 2026-10-15
• Claude systematically debunks the speculative EDAR-selenium hypothesis about East Asians — teortaxesTex · 2026-10-15
• Microsoft's AutoEnvScaling lets agents build their own RL environments, lifting Qwen3 Terminal-Bench from 41.1 to 53.3 — omarsar0 · 2026-10-15

Research channel →