Cliff: Learning Process Rewards from the First Mistake
Peixuan Han, Runhui Wang, Ketan Ramaneti, Jie Hao, Gerald Friedland, Chris Kong
cs.LG
2026-09-03
An off-the-shelf teacher marks the first mistake in a rollout and splits advantages at that cut. Across 12 settings Cliff beats GRPO by 7% on average.
RL with verifiable rewards is the default post-training path for reasoning, but the reward only sees the final answer. A near-complete solution that slips at the last step gets the same zero as a trajectory that was wrong from the first line. Process reward models need a separate scorer and are easy to hack. On-policy distillation wants a same-family teacher with similar reasoning style.
Cliff starts from a coarser observation. Once a chain of thought first goes wrong, later steps are conditioned on an invalid prefix, so scoring them in detail adds little. The useful cut may simply be: where did it first fail.
Cliff is reward shaping on top of GRPO. No extra process-reward model. The teacher solves the problem on its own. The automatic verifier must accept that solution, or the group falls back to vanilla GRPO. The teacher then judges each student rollout against its verified reference, and if the student is wrong it marks the first genuine reasoning error, the Pitfall Step p(a). Overlong rollouts get p(a)=0 so length hacking cannot help.
GRPO still produces two outcome advantages: Acor for correct rollouts, Ainc for incorrect ones. Cliff then splits tokens at the cut. Correct rollouts stay near Acor. Incorrect rollouts get λAcor before p(a) and Ainc after, minus an offset b that recenters the group to zero mean. The main runs use λ=0: the valid prefix of a failed rollout is not positively reinforced, it is merely punished less than the suffix. At λ=1, mean length jumps from 1506 to 1959 tokens and average accuracy falls from 65.66 to 63.98.
The teacher need not share a tokenizer with the student. Judging is easier than solving. Qwen3-32B solves only 65% of the probe set, but with a verified reference it judges student correctness at 91% and lands within about three sentences of human pitfall labels.
Students are Qwen3-4B (SFT on OpenThoughts first) and Phi-4-mini-Instruct. Teachers are an unnamed frontier model, Qwen3-32B, and Gemma3-27B. Math training uses DAPO-math; code training uses Deepcoder with a binary pass-all-tests reward. Baselines are GRPO, GRPO with a teacher that still assigns one advantage to the whole rollout, SFT distillation, and on-policy distillation. Across 12 settings the paper reports a 15% lift over OPD and 7% over GRPO.
| Student | Split | GRPO | Cliff (SOTA teacher) |
| Qwen3-4B | math average | 61.68 | 65.66 |
| Qwen3-4B | code average | 24.20 | 25.96 |
| Phi-4-mini | math average | 49.78 | 51.73 |
| Phi-4-mini | code average | 21.72 | 24.93 |
On Qwen3-4B math, the SOTA teacher moves MATH-500 from 79.00 to 83.20 and AIME from 32.01 to 36.98. Qwen3-32B still reaches 64.62; Gemma3-27B reaches 63.70. Using the teacher only as an outcome judge, without the cut, barely beats GRPO, so credit assignment is the actual ingredient. Dropping the ground-truth filter barely hurts the SOTA teacher (65.66 to 64.90) and costs weaker teachers about two points, while they still often beat vanilla GRPO.
Process supervision here is cheap: an off-the-shelf chat model, no PRM, no shared vocabulary. Teams already running GRPO change how advantages are sliced, not the optimizer. Modest teachers work if a verifier first drops the teacher's own wrong solutions.
This fixes coarse credit assignment. It does not create a verifier for unverifiable tasks.
λ=0 means the correct prefix of a failed rollout never gets a true positive reward, only a smaller negative, so the process signal is weaker than the slogan. Teacher-verifier agreement sits around 85–90%, and a non-trivial slice is teacher-wrong / verifier-right, which may suppress lucky guesses or punish alternate valid proofs. Human-teacher pitfall distance averages about three sentences, so the cut is noisy. Setting p(a)=0 on overlong outputs is blunt. Evaluation is math and contest coding; agent settings are left as future work. The SOTA teacher is unnamed, which makes the strongest curve hard to reproduce.