PACT: From Credit Assignment to Critic Alignment
Jiayan Fu, Hang Xu, Yong Zhang, Zhaokai Luo, Yao Hu, Dongyan Zhao, Mu Chuan
cs.LG, cs.AI
2026-09-22
Three axioms uniquely identify token-level credit. PACT's Actor-then-Critic plus BCE hits 72.87% Avg@16 on four math tests, 8.80 points over GRPO, and 67.4% on SWE-bench Verified.
LLM reinforcement learning usually reveals one outcome reward at the end of a trajectory, then updates every generated token. There is no agreed mathematical definition of how that outcome should be attributed. PPO builds advantages with GAE, GRPO and RLOO use group baselines from several samples of the same prompt, and On-Policy Distillation treats a teacher's log-density ratio as a dense signal. All of these look like credit assignment and do not match.
Long horizons make the mismatch worse. When the outcome reward is bounded, most true token credits are tiny, so intermediate critic error can swamp the signal. DeepSeek-R1's finding that PPO with λ=1 beats λ=0.95 fits this picture. Standard PPO also trains the critic one policy behind: advantages for the current batch come from a critic fitted on the previous policy, and after the actor steps, the critic lags again.
The paper pins token-level credit down with three regularity conditions. Completeness: assigned credits sum to the final reward minus the initial prediction. Prefix Consistency: accumulated credit on a shared prefix does not get rewritten by later tokens or observations. Neutrality: expected credit of the next token is zero given current information. These three uniquely identify credit as the martingale difference Ci = E[R | Fi] − E[R | F{i-1}]. Coarser schemes, including turn-level credit, are just sums of consecutive token credits.
Under an ideal teacher, OPD's expected policy gradient is proportional to the unique credit, so the teacher is an implicit critic. RLOO's response-level baseline is coarser, yet its expected policy-gradient contribution matches token-level credit, usually with more variance. For rewards in [0,1], the expected sum of squared credits is at most 1/4, so most local increments are small. GAE with λ<1 retains intermediate critic errors in the advantage; λ=1 leaves only the prefix value error. PACT therefore sets λ=1.
Policy Aligned Critic Training changes two pieces. The critic is a sigmoid, trained with binary cross-entropy instead of MSE; the optimal prediction is still the conditional mean, with better endpoint behavior. The update order becomes Actor-then-Critic: the current critic first supports the actor step, then one extra forward pass on the same rollout produces importance ratios, and the critic is trained on the corrected target It R so it ends the iteration aligned with the updated actor. In practice the method uses a detached current-token ratio and drops tokens outside [0, 6], avoiding a high-variance product over long replies.
Math training uses Qwen3.5-4B with the OpenCode agent and final-answer correctness on 3,200 DAPO-Math-17k problems. Avg@16:
| method | AIME 2025 | AIME 2026 | BeyondAIME | HMMT Nov. 2025 | average |
| Base | 46.67 | 51.25 | 28.63 | 37.50 | 41.01 |
| GRPO | 76.50 | 74.78 | 41.81 | 63.19 | 64.07 |
| PPO (λ=1.0) | 66.04 | 73.96 | 40.31 | 58.54 | 59.71 |
| PACT w/o IS | 76.04 | 82.50 | 51.38 | 61.04 | 67.74 |
| PACT | 83.12 | 85.21 | 51.69 | 71.46 | 72.87 |
PACT averages 72.87%, 8.80 points above GRPO, 13.16 above PPO (λ=1.0), and 21.73 above SAO. PPO with λ=0.95 averages 26.31% and collapses. Removing importance correction on the critic target drops the average to 67.74%, worse on all four benchmarks.
On SWE-bench Verified with Qwen3.6-35B-A3B, Pass@1 is 60.8 (base), 65.4 (GRPO), 65.0 (PPO), 63.6 (SAO), and 67.4 (PACT). In a fixed-policy critic pretraining study, BCE converges faster than MSE and separates successful from failed trajectories more clearly.
Group baselines drop the critic and pay in response-level noise. The uniqueness theorem says what token-level credit is, and shows that OPD's teacher and RLOO's baseline are approximations of the same object. Actor-then-Critic costs one extra forward pass and no extra rollouts. For long-horizon agentic RL that still uses a value head, λ=1 plus BCE plus a cheap IS correction is a more defensible default than retuning λ=0.95.
Uniqueness holds only under the three regularity conditions; other axioms yield other credits. The object is statistical, not the causal effect of swapping a token. The paper does not produce an exact practical estimator. Single-token importance ratios with clipping can still bias long replies. Each domain is a single model and harness, with no cross-family replication.