CoRT: Counterfactual Replay for Token-Level Rubric-Guided Policy Optimization
Bo-Wen Zhang, Junwei He, Wen Wang, Song-Lin Lv, Wentao Ma, Rongyi Lin, Shuhan Zhong, Lan-Zhe Guo
cs.AI
2026-07-28
Rubric GRPO spreads one reward across all tokens. CoRT rescores the answer under two prompts and uses the contrast to assign credit token by token, +4.4 points on average.
Rubric-based RL evaluates model outputs against an explicit rubric, which carries more signal than a scalar reward. But once it's plugged into a GRPO-style pipeline, it gets flattened: the judgment made against the rubric for a whole response is compressed into one scalar response-level reward, turned into one response-level advantage, and broadcast uniformly to every token. The problem is that different rubric criteria usually anchor on different spans, formatting choices, or semantic decisions; whether one span is right and whether another is right should be credited separately, and uniform spreading smooths the signal right back out.
CoRT (Counterfactual Replay for token-level credit) redistributes that uniform advantage across tokens without touching the response-level reward. It doesn't train an auxiliary token scorer; it uses counterfactual replay.
For one sampled response, it recomputes each token's log-likelihood under two prompts. One is the original rubric-conditioned prompt; the other is a matched, criteria-free prompt. The per-token likelihood contrast delta is a proxy for how much that token depended on the rubric context: a token whose likelihood is markedly higher when the criterion is present was generated because of that criterion and should get more credit.
The contrasts pass through a sigmoid into bounded replay-margin scores, then a SmoothStep schedule normalizes them within each response (weights averaging to 1), and the result multiplies the signed GRPO advantage to redistribute it token by token. No auxiliary scorer, no change to the response-level reward; the simplicity and stability of GRPO are preserved.
Across Qwen3-4B-Instruct, Qwen2.5-7B-Instruct, and Qwen3-14B, and four instruction-following benchmarks (IFBench, IFEval, MultiDimIF, AdvancedIF), CoRT beats the matched response-level GRPO in the vast majority of configs, by an average of 4.4 percentage points. Per benchmark: +6.10 on MultiDimIF, +3.16 on Rubric, +2.81 on IFBench Instruction. It matches a baseline that learns a separate token-level relevance model, but without that relevance-learning stage. It also plugs into DAPO and GSPO, so it's a credit-assignment module rather than something welded to GRPO.
For instruction-following RL, this is a low-friction way to push a coarse signal finer without redefining the reward: no auxiliary model, just contrasts the model can already compute. It reclaims the part of the rubric signal that GRPO was throwing away.
The authors concede the ceiling tracks signal strength: when a criterion barely moves token likelihoods, when it's redundant with the original prompt, or when the response-level reward is noisy, the counterfactual contrast is weak and so is the gain. It's tested only on tasks with explicit rubrics. They point to finer-grained interventions, criterion-specific allocation, and longer-horizon credit as future work. One implicit cost the paper leaves unquantified: recomputing likelihoods twice per response clearly raises per-step compute, and that overhead isn't priced.