MetaRubric: Learning to Reward for Rubric-Based Reinforcement Learning
Yuxuan Fan, Jaehong Yoon
cs.AI
2026-10-02
MetaRubric takes the min of satisfaction, coverage, and support, then adapts rubrics to policy errors. It beats static-judge GRPO by 6.0-20.4 PubMedQA points on three backbones.
Rubric-based RL scores open-ended answers criterion by criterion, then optimizes the policy on the weighted sum. The scheme assumes a high score means the required information or action is actually in the response.
That assumption fails often enough to matter. The paper names the failure Vacuous Credit: a judge marks a criterion satisfied when the response never supplies what it asks for. In the stent-checkup example, the rubric says to ask where the patient lives. The model only notes that guidelines differ by region, gives no visit interval, and never asks for a location. The judge still awards the full 8 points.
The error shows up in the training curve. Qwen3-8B trained with GRPO on HealthBench sees its training reward rise, while a held-out panel (GPT-5.5, Gemini 3.5 Flash, and DeepSeek-V4-Pro, unanimous agreement required) records falling rubric satisfaction. A deletion test makes the same point on 1,000 rollouts judged correct during training. After GPT-5.5 removes the content required by the target criterion, GPT-4o-mini keeps 82.0% of those awards and GPT-5.4-mini keeps 69.8%. Deleting only generic filler leaves 93.1% and 95.0%.
GRPO advantages are computed inside a rollout group. One vacuous award can flip a response's advantage from negative to positive, so the update reinforces a safe non-answer.
MetaRubric alternates an inner policy loop with an outer rubric loop. The inner loop runs GRPO under a frozen rubric. The outer loop then rewrites weights and wording from that stage's responses. Every prompt is paired with a counterfactual that changes one task fact, such as age 40 to 60. Each response is scored only on its own rubric, so a fact-dependent requirement has to change with the fact.
For each criterion the judge returns three scores in [0, 1]: satisfaction, coverage (is the information or action present), and evidential support (is that claim backed by the response and the evidence the task allows). The criterion score is the minimum. Stating the wrong number can still cover the slot; satisfaction and support pull the score down. Two response-level gates sit on top. A support cap stops a few credited criteria from hiding unsupported claims elsewhere. A quality factor, the minimum of relevance, coherence, and concision, down-weights padded or off-topic answers.
An auxiliary QA reward does not depend on how a criterion is worded. Facts from the reference are turned into four-option questions, questions the reader can solve without the response are dropped, and the bank is frozen before training. Qwen3-1.7B answers from the policy response alone. The training reward mixes rubric and QA scores with coefficients 1.0 and 0.3. Original and counterfactual prompts share one GRPO group. PubMedQA already has yes/no/maybe labels, so it skips auxiliary questions.
At each stage boundary, weights move first. Criteria are grouped by severity and by how they correspond across the counterfactual pair. The four severities, nice-to-have through contraindication, use base weights 4, 5, 6, and 8. Groups that fail more than average gain log-weight with step size 0.1. Severity order cannot reverse, each adjustment stays within log(6/5) (about 0.18), and adjustments are centered at zero, so a frequently missed nice-to-have cannot outrank a must-have. Wording changes next, at most one criterion per round, anchored to the prompt's initial rubric. A revision is kept only if agreement with a reference panel on held-out responses rises by more than 0.02 and a separate semantic review accepts it. Weights carry forward either way.
The benchmarks are PubMedQA, HealthBench-Hard, and two panoramic dental tasks, MMOral-X and MMOral-OPG. Backbones are Qwen3-4B, Qwen3-8B (Qwen3-VL at the same size for images), and Gemma-e2b. Baselines are static-judge GRPO, DAPO, Dr. GRPO, and GSPO. GPT-5.5 grades every model-based test metric. HealthBench-Hard reports an accuracy-axis score; PubMedQA reports answer accuracy. MMOral-X is the mean of three difficulty subsets, and MMOral-OPG is a single overall score.
| Backbone | PubMedQA | HealthBench-Hard | MMOral-X | MMOral-OPG |
| Qwen3-4B / GRPO | 72.40 | 10.56 | 5.72 | 23.18 |
| Qwen3-4B / MetaRubric | 78.40 | 13.02 | 7.73 | 26.35 |
| Qwen3-8B / GRPO | 70.00 | 7.15 | 7.08 | 27.35 |
| Qwen3-8B / MetaRubric | 76.80 | 10.34 | 9.21 | 30.44 |
| Gemma-e2b / GRPO | 51.60 | 13.07 | 4.74 | 31.28 |
| Gemma-e2b / MetaRubric | 72.00 | 15.68 | 6.03 | 35.10 |
Against static GRPO, PubMedQA rises by 6.00, 6.80, and 20.40 points, HealthBench-Hard by 2.46 to 3.19, MMOral-X by 1.29 to 2.13, and MMOral-OPG by 3.09 to 3.82. Untrained PubMedQA scores are 52.60, 44.20, and 48.00. GRPO already lifts the Qwen models to around 70; on Gemma-e2b it stalls at 51.60, and MetaRubric supplies most of the gain.
MetaRubric is best in 19 of 21 model-family by metric comparisons. Dr. GRPO is slightly higher on Qwen3-4B HealthBench-Hard (13.28 vs 13.02) and Gemma-e2b MMOral-OPG (35.42 vs 35.10). Auxiliary QA accuracy on the open-ended benchmarks beats GRPO by 3.23 to 5.68 points.
On the Qwen3-4B ablations, freezing both descriptions and weights costs 3.17 points on HealthBench-Hard and 4.40 on MMOral-OPG. Dropping the evidence checks costs 1.22 and 2.59. Revision matters more on the first task, weight changes on the second.
Retraining from the final rubric alone moves HealthBench-Hard from 8.83 to 9.04 and MMOral-OPG from 20.59 to 21.78, against 13.02 and 26.35 for the full method. That is 5.0% and 20.7% of the gain. Criteria whose description or correspondence label changed at least once are 45.2% of HealthBench, 29.8% of PubMedQA, and 60.6% of MMOral-RL. Mean length grows by 5.5, 3.9, and 5.6 words.
A rising reward is not evidence that the rubric is being met. Vacuous Credit persists through training even with GPT-5.4-mini as the training judge.
Capping each criterion by coverage and support blocks full credit for empty assurances. The rubric can still move with the policy's errors, but the initial wording is an anchor, agreement must clear 0.02, and log-weight shifts stay near 0.18. Retraining on the final rubric recovers only a small slice of the gain, so a one-shot rewrite is not a substitute for the alternation. The bill is real: three-way judging, counterfactual pairs, and a review step at every stage boundary.
The lift depends on the task. PubMedQA gains as much as 20.4 points over GRPO. Open medical dialogue and dental image tasks mostly move 1 to 4 points, and HealthBench-Hard still sits near 10 to 16. The reward tracks evidence more closely. Absolute answer quality on these medical benchmarks stays limited.
The appendix states two limits: one seed per configuration, with no run-to-run variance, and no matched comparison to related methods whose training code is unreleased.
All four benchmarks are medical text or dental images. Open-ended writing, tool use, and general dialogue are untested. The main table uses GPT-5.5 as the test judge and GPT-5.4-mini for training and the outer loop. Whether the ranking survives a different judge family is not reported.
That one vacuous award can flip an advantage sign is plausible under group normalization, but the paper gives no rate. The deletion set is 1,000 rollouts already marked correct, so the retention rates do not describe every sample. Auxiliary QA is part of the training objective, so its rise is not an independent quality check. Faithfulness of the single-fact counterfactuals is not audited on its own. Weight updates are clipped near 0.18, and revision versus reweighting trade places across the two ablated tasks. Nothing shows that split will hold in another domain.