Rubric Rewards from Item Response Theory
Milad Yazdani, Yaser Souri, Xiren Zhou, Pranit Chawla, Dena Shahriari, Subhojit Som, Xia Song
cs.CL
2026-09-29
RRT treats rubric verdicts as IRT test items and infers rollout quality as the GRPO reward: +1.7 macro points over Vanilla GRPO, within 0.1 of full judging at half the budget.
RL on language models is easy when the reward is machine-checkable: code passes tests, the math answer matches. Medical advice, long-form writing, and open-ended QA have no such signal. The standard recipe generates a rubric per prompt, has an LLM judge issue binary verdicts criterion by criterion, and sums the satisfied criteria's points into a scalar reward for GRPO.
That sum fails two ways. Distinct verdict patterns receive identical totals: with five equally weighted criteria, passing four easy ones while failing the hardest and failing the easiest while passing four hard ones both score 0.8. Since GRPO computes advantages by centering rewards within a group of rollouts for the same prompt, tied rewards are judge money spent for zero gradient. And fixed points encode how much the rubric author thinks each criterion should count, not how strongly its verdict separates the current rollouts. Judge cost adds a third pain: every rollout is judged on every criterion, so requests grow linearly with rubric size.
RRT (Rubric Response Theory) imports IRT (Item Response Theory), the measurement framework behind standardized testing: each item carries difficulty and discrimination parameters, each examinee an ability, and the probability of a correct answer follows from the three. Criteria become items; rollouts become examinees.
A theorem ties it together: under this model the likelihood score maximizes the local signal-to-noise ratio for quality among all scalar functions of the verdicts, reaching the Fisher information bound. Once two criteria have different parameters, no fixed points vector attains the bound at every quality level.
Default setup: Qwen3.5-4B policy, GPT-5.5 with reasoning disabled as judge, four datasets (Medical, Science, RaR Science, RubricBench).
| Comparison | Metric | Result |
| RRT + online RPN vs Vanilla GRPO | macro criterion score, 4 datasets | +1.7 points |
| RRT vs POW3R / DIVA | same | +0.8 points; highest or tied in every column of Table 2 |
| RRT vs Vanilla GRPO | difficulty bands, Medical + Science | leads in 7 of 8 bands; +2.8 to 5.6 on hard and very hard |
| RRT vs point-sum reward | tied rollout pairs within groups, 12 cells | 2% to 58% fewer ties |
| RRT vs mean of other verdicts | leave-one-criterion-out prediction, pooled ROC-AUC | 79.5 vs 69.3 |
| Adaptive Fisher, budget 0.5 | macro score vs Vanilla GRPO with full judging | within 0.1 points; judging time down 49.1% to 49.6%, step time down 23.5% to 28.7% |
Transfer results are more modest: the macro criterion score is 0.2 points below Vanilla GRPO on Qwen3.5-2B and 0.1 above on Llama-3.1-8B-Instruct, essentially parity; gains on HealthBench and ResearchQA run 0.1 to 0.7 points. A side finding: RRT-trained policies produce shorter responses, with dataset medians 10.6% to 47.9% below the point-sum baseline.
Three payoffs for post-training teams. Reward resolution: tied rewards in GRPO are judge spend with no gradient attached, and RRT recovers that signal exactly where the base policy struggles, on hard criteria. Judge cost: judging was the longest measured stage of each policy step, and cutting the criterion budget in half at unchanged scores converts directly into training throughput. Engineering footprint: the RPN, EM loop, and selector exist only during training; the deployed policy gains zero parameters.
The framing may outlast the numbers. POW3R and DIVA stay on the road of learning one weight per criterion inside a weighted sum. RRT swaps aggregation for measurement: effective weights vary with quality level and are pinned down by verdict data plus criterion text.
The model assumes every criterion is a monotone indicator of one shared quality (unidimensionality) and that verdicts are independent given quality (local independence). Rubrics with explicit tradeoffs, such as demanding both brevity and detail, break the first assumption. The paper's own diagnostics: fitted quality explains 72.8% to 85.3% of pairwise mutual information, but similar-text criteria retain residual redundancy, 3.4 to 11.9 points above a bootstrap null; the authors cite the known bias of unidimensional IRT (Yen, 1984) and leave a multidimensional extension to future work.
Caution flags beyond the self-reported ones: the +1.7 headline comes from one policy (Qwen3.5-4B), with parity on 2B and 8B; each condition has a single training run, so seed-level variance is uncovered (the confidence intervals fix the trained policies and resample prompt groups); RPN text-based difficulty prediction sits below half rank correlation, so gains lean on online calibration staying stable; and the judge is GPT-5.5 with just 1.27% disagreement from consensus. Whether the inference survives a weaker, noisier judge is untested.