Enhancing Rubric-based RL via Self-Distillation
Mingxuan Xia, Yuhang Yang, Chao Ye, Shuai Zhu, Shenzhi Yang, Guangcheng Zhu, Yuhang Zhang, Cheng Peng, Haobo Wang, Siqing Wang
cs.LG, cs.AI
2026-07-20
CriPO layers on-policy self-distillation onto GRPO to fix both unexplored and suppressed criteria, beating GRPO on medicine/science benchmarks while halving training steps.
Rubric-based RL trains language models on open-ended tasks (medical Q&A, research Q&A) by scoring each rollout against a rubric: a set of weighted criteria cj scored by a judge model, then aggregated into a single scalar reward for GRPO. Richer than a binary reward, but the authors show it has two failure modes, and prior work has fixed one while missing the other.
The first is Unexplored Criteria: when no rollout in a sample group satisfies a criterion, that criterion gets no gradient. Methods like RuscaRL and HeRL handle this by injecting rubric criteria as external scaffolding during rollout generation. That creates a train-inference mismatch: the policy is trained on rollouts produced with guidance that is absent at inference, and the error accumulates through autoregressive decoding.
The second, which this paper names Suppressed Criteria, is new. A criterion is satisfied by some rollouts, but scalar reward aggregation drowns its contribution and assigns it a non-positive advantage, so the model is pushed away from the very behavior that satisfies it. The authors measure this directly: across training of Qwen3-4B on RaR-Medicine, over 57% of samples contain suppressed criteria, averaging 1.8 per sample.
CriPO (Criterion-Distilled Policy Optimization) layers on-policy self-distillation onto the GRPO backbone. Teacher and student are the same model, differentiated only by input conditioning, so there is no train-inference mismatch.
For unexplored criteria it builds a "criterion-injection self-teacher." It picks the highest-advantage rollout y from a group and conditions the model on the unexplored criteria 𝒞u to act as teacher. Guided by criteria it would not spontaneously satisfy, the teacher skews toward criterion-satisfying tokens. Distillation then happens only at tokens where teacher and student disagree, via forward-KL. This localization is the key move: a contribution filter keeps only the tokens that account for 95% (γ=0.95) of cumulative KL, which in practice is just 34.6% of all tokens. The missing behavior is injected precisely, not by copying the teacher wholesale.
For suppressed criteria it builds a "counterfactual self-teacher." For negative-advantage rollouts that nonetheless satisfy some suppressed criteria, it locates criterion-relevant tokens where the counterfactual teacher diverges from the student in log-probability and where the teacher is itself unsure (threshold α=0.1). At those tokens it flips the advantage from negative to positive (τflip=0.1), rescuing genuinely useful tokens that GRPO would otherwise suppress along with the rest of the rollout.
The final objective is ℒCriPO = ℒGRPO + β·ℒOPSD, with β = 0.05 for Qwen3-1.7B and 0.03 for Qwen3-4B.
Training uses RaR-Medicine (15,658 samples) and RaR-Science (10,874); cross-domain tests cover HealthBench, LLMEval-Med, and ResearchQA. The training judge is Qwen3-32B; the main eval judge is GPT-4o-mini, with Qwen3-32B used again to check robustness.
Main results (GPT-4o-mini, average %):
| Method | Qwen3-1.7B | Qwen3-4B |
| Base | 57.2 | 63.7 |
| GRPO | 59.2 | 68.2 |
| HeRL | 60.4 | 68.4 |
| OPSD (naive distillation) | 38.2 | 43.6 |
| CriPO (this paper) | 62.4 | 69.6 |
CriPO tops both scales, beating GRPO by 1.4 to 3.2 points. The OPSD collapse is the telling comparison: it shares CriPO's premise (distill rubric criteria into the policy) but skips localization and counterfactual targeting, and on 1.7B it drops from the base's 57.2 to 38.2. That confirms the train-inference mismatch can wreck the model on its own, and that CriPO's precise, minimal injection is load-bearing.
With Qwen3-32B as eval judge the trend holds: CriPO 47.1 vs GRPO 43.7 at 1.7B, and 61.5 vs 59.4 at 4B.
On efficiency, CriPO reaches GRPO's converged best at roughly step 175, about 2× fewer optimization steps, then keeps climbing to a higher final score. Entropy stays higher throughout training and responses grow longer, which the authors attribute to broader exploration.
For anyone doing RLHF or RLVR, two things stand out. First, the paper quantifies a loss that scalar reward aggregation was quietly eating: 57% of samples and 1.8 suppressed criteria each. That is a real, previously underweighted cost. Second, it offers a fix that does not introduce train-inference mismatch, and it is modular. The two interventions (CriPO-U, CriPO-S) work alone, and the ablation shows each suppresses only its own failure mode; combined, they take the best of both.
The authors concede one dynamic failure: as the model learns to satisfy more criteria during training, the count of suppressed criteria actually rises (broader exploration means more criteria get drowned out). CriPO keeps that count below GRPO's, but it does not eliminate the effect, which is partly a moving target.
Other concerns are harder to dismiss. Every gain is measured by LLM judges with no ground-truth reward, so part of the lift is alignment with the judge's preferences rather than pure capability. On 4B, CriPO beats GRPO by only 1.4 points, a modest absolute margin. The two interventions each carry a cluster of hyperparameters (γ, α, τflip, β, top-K=3), and the paper reports no sensitivity analysis, so it is unclear whether they transfer without retuning. Validation is confined to medicine and science Q&A; behavior on tasks with verifiable answers, like math or code, is unknown.