RUC's TIPS: outcome-only RL trains PRMs, hitting 85.2 F1 on ProcessBench with 3.2K trajectories

RUC · hf · 2026-10-12

Problem

Process reward models (PRMs) are key for LLM post-training and test-time reasoning, but training strong PRMs is costly: human step annotation doesn't scale, and Monte Carlo estimation is expensive and can drift from steps' intrinsic correctness.

TIPS method

Researchers at Renmin University introduce TIPS (Thinking-Induced Process Supervision), an outcome-only RL framework for training generative PRMs: the model emits a CoT followed by step-level labels and an outcome label; reward depends solely on whether the predicted outcome matches ground truth, with group-relative advantage optimizing the whole response. Intuition: when checking intermediate steps helps determine the outcome, more accurate step checks yield better outcome judgments and higher rewards — reinforcing step-level verification without explicit process supervision.

Results

• Validated on math and agent benchmarks across four backbone families
• TIPS-Qwen3-4B-Thinking-2507 reaches 85.2 F1 on ProcessBench with only 3.2K outcome-labeled trajectories, surpassing all evaluated trained PRMs and prompt-only judges like GPT-5.4-Instruct and Claude-4.7-Opus, trailing only o1-mini
• Code and data open-sourced

Original post →

More from Research

• 1,200 AI answers show Reddit citations are nearly all from Google engines: Gemini 52.4%, ChatGPT 0% — drrickio · 2026-10-12
• Open experiment log pushes integer multiplication bounds down to κ≈0.001467, invites agent collaboration — IgorCarron · 2026-10-12
• LLMs don't guess, they optimize: a statistical mechanics take claims 3,418 gradient steps to coherence — TinfoilTricorn · 2026-10-12
• A critical look at Lean4: why no one can prove the proof assistant is consistent — thomasahle · 2026-10-12
• ThinkDrag: mmlab-ntu adds visual reasoning to drag-based image editing, claims SOTA — mmlab-ntu · 2026-10-12
• Dex-One2Many learns dexterous manipulation from one human video, +71% on unseen setups — Jusuk Lee · 2026-10-12

Research channel →