Not All Tokens Deserve Equal Credit: Counterfactual Sensitivity Credit Reallocation for Long-CoT Reasoning
Qiangqiang He, Zhongheng Wu, ZiJian Wang
cs.AI
2026-07-30
Privileged self-distillation shifts in GRPO reflect counterfactual sensitivity, not answer signal. Downweighting sensitive tokens lifts 5 long-CoT math benchmarks by up to 8 points.
Critic-free RL methods like GRPO turn a whole response's reward into a single advantage and spread it evenly across every token. In a thousand-token chain of thought, the step that actually decides correctness, the dead-end branch, and the continuation already determined by the prefix all get the same credit. OPSD (on-policy self-distillation) tried to fix this by fitting a student policy to a privileged teacher that has been told whether the answer is right, on the assumption that the teacher's likelihood shifts carry token-level signal pointing toward the answer.
This paper tests that assumption and finds it false.
The diagnosis is a counterfactual probe. For one problem and one sampled trajectory y, freeze the trajectory and every prefix, change only whether the model is told the answer is right or wrong, and watch each token's log-probability move. The two opposing conditions are c^pos ("the verifier judged this solution correct, wonderful") and c^neg ("judged incorrect, terrible").
If OPSD's premise held, opposing conditions would push tokens in opposite directions, and the tokens carrying the reasoning would move the most. Neither is true:
So a large privileged shift marks a token that is easy to paraphrase, not a token that matters to the answer. It measures counterfactual sensitivity, not learning value.
CSCR (Counterfactual Sensitivity Credit Reallocation) is a minimal change to GRPO built on this. Take each token's sensitivity s = max(|z^pos|, |z^neg|). Tokens above a threshold lambda have their credit weight decayed exponentially toward 1 minus gamma; tokens below stay at 1. Renormalize within the trajectory so the total credit budget and the verifier-determined direction are preserved. Credit moves from sensitive surface tokens to insensitive content tokens.
Qwen3-1.7B and Qwen3-4B, trained 300 steps on DAPO-17K, Mean@32 on five competition benchmarks:
| Model | Method | AMC23 | AIME24 | AIME25 | AIME26 | SMT25 |
| Qwen3-1.7B | GRPO | 84.1 | 47.4 | 36.7 | 37.4 | 39.5 |
| Qwen3-1.7B | CSCR | 88.8 | 50.1 | 43.6 | 42.2 | 44.0 |
| Qwen3-4B | GRPO | 96.9 | 71.5 | 65.0 | 65.3 | 57.4 |
| Qwen3-4B | CSCR | 97.8 | 74.4 | 68.3 | 67.2 | 59.0 |
The 1.7B model gains about 6 points on average (AIME25 +8.1 is the largest single gain); the 4B model gains about 3.7. Same number of policy updates, higher reward throughout, and it does not cheat by shortening: response length drifts slightly up.
The ablations nail the diagnosis. Flip it and use the shift sign to direct tokens (SD-GRPO): training collapses within 30 steps, reward to zero, entropy to zero, responses inflate to the 20,480-token limit and degenerate. On modulation strength, mild downweighting (gamma=0.2) is the most stable; upweighting or strong downweighting (gamma=0.5, 1.0) destabilizes optimization or collapses responses to 2,000–6,000 tokens.
For anyone doing long-CoT RL, this is a cheap, plug-in GRPO patch: a weight multiplied into the advantage before optimization, minimal code. The more valuable part is the diagnosis. A lot of "dense token-level supervision" intuition (self-distillation, token-level credit) assumes a token's probability shift equals its contribution to the answer. This paper shows that in long-CoT math at least, the tokens that shift most are style, not substance.
Only math reasoning is tested; transfer to code or general reasoning is not shown. CSCR re-scores every sampled trajectory under two extra conditions, roughly two extra forward passes over vanilla GRPO, and the paper does not quantify that overhead. The threshold lambda and strength gamma both need tuning; the authors admit strong settings collapse, so a new model or dataset likely means retuning. An open question they leave: some genuinely useful reasoning tokens may be downweighted because they also happen to be highly substitutable.