Rubric Dropout: a one-line fix that lifts true quality up to 7 points in rubric-rewarded RL

Rubric Dropout: A Simple Way to Mitigate Reward Hacking in Rubric-as-Reward RL

Minglai Yang, Xinyu Guo, Utkarsh Tyagi, Mian Zhang, Razvan Dumitru, Sunjie Hou, Yunzhong He, Daniel Yue Zhang, Ying Liu

cs.LG, cs.AI, cs.CL

2026-08-12

Rubric-rewarded GRPO hacks the reward: proxy climbs while real quality falls. Dropping criteria each step lifts true quality at every checkpoint (up to +7 pts), no cost.

What problem this solves

Post-training a language model with an LLM judge that grades against a rubric, a checklist of criteria, is now standard for tasks with no single right answer, medical Q&A and research synthesis. Algorithms like GRPO optimize that rubric score directly. But a rubric is a fixed proxy for quality, not quality itself. Train against it long enough and the policy finds shortcuts: it maxes out every criterion while real quality stalls or drops. This is Goodhart's law showing up again in RL post-training, and the paper calls it reward hacking.

The authors measure it head-on. They train Qwen3-8B with GRPO on medical and science rubrics, and grade out-of-distribution eval sets with both the training judge (gpt-4o-mini) and a stronger cross-family gold judge (claude-sonnet-4-6). The two curves diverge: the training judge's score climbs to 72%, while the gold judge's score peaks and then falls, by 3 points on HealthBench-Hard and by 22 points on ResearchQA. A judge with a fixed bias would shift the gold curve by a constant, not push it down while the training score rises. So the gap is hacking, not noise.

Method

Rubric Dropout is neuron dropout moved onto the reward. At each training step, a random fraction f of the rubric's positive-weight criteria is dropped before the reward is computed. The policy never optimizes the same full rubric twice, so it cannot reliably exploit any single fixed criterion.

Three design choices matter:

• One mask per rollout group. GRPO compares multiple responses to the same prompt and computes group-relative advantages. If each response dropped a different subset, they would be graded on different sub-rubrics and the advantages would not be comparable, corrupting the gradient. So the mask is seeded by SHA256(instanceid, step) and shared across the whole group, with no cross-worker communication.
• Evaluation always uses the full rubric. Dropout is train-only; eval and the gold judge always score against all criteria.
• Safety-critical criteria are protected and never dropped.

The theory (Appendix A): because the mask is shared by the whole group, any reward normalizer that depends only on the mask cancels in GRPO's standardized advantage, so normalization is not a knob to tune. Dropout's real effect is variance injection. The variance is largest when a response's advantage rests on a single high-weight criterion, and smallest when the response is broadly better than its group. That is the same anti-co-adaptation logic as neuron dropout, transplanted onto the reward.

Results

The main comparison is 8B, base (no dropout) versus f=30% and f=50%, averaged over the window after hacking sets in (steps 400-600):

Across 11 matched checkpoints in the window, both dropout runs beat base on gold at every point, cut both hacking measures (proxy-versus-gold gap and overclaim rate), and keep in-domain training reward at 97% or higher. The 4B results agree (medical f=50% +3.0, science f=30% +5.3).

One detail matters more than the averages: peak capability is tied. Across the sweep (20%-50%), best-checkpoint gold sits at 30.6%-31.5% against base's 31.2%, essentially the same. The differences are all in how far the score falls after the peak. Dropout does not make the model smarter; it stops it from getting dumber the longer it trains.

The fraction sweep (medical, 8B): 20% gives +0.6, 30% +1.0, 40% +0.4, 50% +2.0, and 60% flips to -0.5. The 30%-50% band is a wide safe range; the hyperparameter is not delicate. At 60% the kept sub-rubric no longer covers quality, the expected failure mode.

The most counterintuitive comparison is reweighting (POW3R): instead of dropping, rescale each criterion by its within-group verdict variance. Weighting important criteria more sounds smarter, but it is the worst run of all. Gold is 27.0% (1.2 below base), overclaim is the highest at 42.2%, and it loses to base at all 11 matched checkpoints. The authors' read: weighting by variance concentrates optimization pressure on exactly the criteria the policy is currently learning to game, amplifying the feedback loop that dropout dilutes.

Why it matters

Rubric-as-reward is close to the default shape of judge-based post-training now; any team doing LLM-graded RLHF or GRPO runs into this. Rubric Dropout's appeal is that it is nearly free in engineering terms: one line of code, one hyperparameter, no extra judge calls, and a forgiving 30%-50% band. For a common and widely ignored failure mode (the model getting worse the longer you train), it is a cheap default guard.

The applicability is also clear-cut: you need a group-relative RL algorithm (GRPO and family) and a rubric made of independently gradeable criteria. Meet those two and it drops in.

Limitations

The authors list four, plus one open question:

• Single seed. Every configuration is one run, because preemptible-only compute ruled out seed replication. The error bars reflect within-run checkpoint variation, not cross-seed variation, so the effect sizes await seed confirmation.
• The gold judge is not ground truth. A stronger judge is still a judge. The claims rest on divergence and on run-to-run comparisons under identical judges, both of which survive a constant judge bias but not a distribution-dependent one.
• In-domain cost is measured on the training set. No in-domain cost means full-rubric reward saturates on training prompts; unseen in-domain prompts were not measured.
• Narrow scope. One model family (Qwen3), two sizes, two domains, one algorithm (GRPO). Generalization to PPO or other families is untested.
• Mechanism is unsettled. The authors concede that anti-co-adaptation and implicit early stopping predict the same figures here. The decisive test is the gold-versus-overclaim frontier at two-plus epochs: separation would establish co-adaptation, continued overlap would mean the gains reduce to early stopping. That matters, because if it is the latter, most of the benefit is available just by picking an earlier checkpoint.

Terms

• rubric: a grading checklist that decomposes a good answer into independently checkable criteria (e.g. "states the correct diagnosis", "mentions key differential diagnoses"), each judged pass/fail by an LLM.
• GRPO (Group Relative Policy Optimization): an RL algorithm that samples multiple responses per prompt and computes advantages relative to the group average, with no separate value model.
• reward hacking: the policy learns shortcuts that raise the reward signal without improving real task quality.
• OOD (out-of-distribution): evaluation data not seen in training, used to test whether the model learned genuine capability rather than training-set artifacts.
• overclaim / underclaim: overclaim is when the training judge passes a criterion that the stronger gold judge fails (an overstatement); underclaim is the reverse. A high overclaim rate is a direct symptom of reward hacking.
• Goodhart's law: when a measure becomes an optimization target, it ceases to be a good measure. This paper is its empirical demonstration in RL post-training.

Source

• arXiv 2608.11669v1
• PDF
• Hugging Face

What people are saying

• Rubric Dropout: A One-Line Fix to Mitigate Reward Hacking in RL burny_tech
• Rubric Dropout: A Simple Way to Mitigate Reward Hacking in RL joecole

Related papers

• IER Shows 0.1%–1% of Tokens Can Match Full On-Policy Distillation
• 4-bit models that keep classification accuracy still swap 14–46% of retrieval top-1s
• Signed KDA gates yield 2D rotations in one update, keeping 96% of KDA throughput
• KernelBench's official check misses 16.9% of witnessed faults, 78.6% of precision bugs
• Regularized harness self-improvement lifts OOD scores up to 4.7 points on 36% fewer tokens
• FAIR's higher: monkey-patch PyTorch modules to backprop through inner-loop optimizers

All paper explainers