Autorubric: A Unifying Framework for Rubric-Based LLM Evaluation on Non-Verifiable Tasks
Delip Rao, Chris Callison-Burch
COLM 2026 camera ready copy
cs.CL, cs.AI
2026-02-13
Penn's Autorubric makes rubric judging an auditable per-criterion stack. Gemini-3-Flash 5-shot hits 80.7% on RiceChem; judge family dwarfs mitigation tweaks on CHARM-100.
Open-ended writing, chat quality, and research synthesis cannot be scored with unit tests, so LLM-as-a-Judge is the default. Those judges still pick up position bias, random self-inconsistency, mixing several quality dimensions in one call, forcing a label when the evidence is thin, and drifting calibration across model families.
A rubric is often treated as a longer prompt. In educational measurement it is a scoring instrument: analytic versus holistic, binary versus ordinal scales, weights, abstention, calibration, and rater reliability. Those choices live in scattered papers and half-finished repos, so every lab rebuilds the same machinery. Delip Rao and Chris Callison-Burch at Penn packaged the design space into Autorubric, an open-source framework (COLM 2026 camera-ready; API snapshot v1.5.3).
The default is an analytic rubric. Each criterion is a separate LLM call that returns MET, UNMET, CANNOTASSESS, or a categorical option, plus a short evidence-citing reason. N judges times M criteria run in parallel.
Only three criterion types are allowed. Binary is met or unmet. Ordinal uses a bounded scale with behavioral anchors. Nominal is unordered classification. Continuous scores are excluded on purpose: models pile onto a middle integer even when asked for decimals. Each option carries an explicit 0-1 value, and option order is shuffled by default so the number is not tied to position.
Weights can be positive or negative; negatives penalize anti-patterns. If any positive weight remains, the score is the weighted sum divided by the positive-weight total, clipped to [0, 1]. A penalty-only rubric starts at 1 and falls. Abstention strategies are SKIP, ZERO, PARTIAL, and FAIL.
Few-shot calibration samples exemplars with balanced verdicts so the judge cannot read a base-rate prior off the prompt. Ensembles aggregate by majority, weighted vote, unanimous, or any-vote. After a run the framework reports Cohen's κ, quadratic-weighted κ, Spearman correlation, and Earth Mover's Distance. EvalRunner handles caching, rate limits, resumable checkpoints, and per-call cost.
RiceChem is 1,240 college chemistry long answers graded on 27 binary criteria. On 819 held-out response-criterion decisions:
| Method | Accuracy |
| GPT-4 zero-shot (prior work) | 70.9% |
| Autorubric + Gemini-3-Flash, 0-shot | 78.0% (639/819) |
| Same setup, 5-shot | 80.7% (661/819) |
| Fine-tuned RoBERTa+MNLI (prior work) | 86.8% |
Five-shot beats zero-shot by 2.7 points (54 flips up, 32 down). The authors treat the gap as descriptive and skip cluster-adjusted tests nested in student responses. The remaining 6.1-point hole versus the fine-tuned classifier is expected for a few-shot judge.
ResearcherBench scores Deep Research systems on 65 questions, 931 weighted binary criteria, and 5,586 judgments. Coverage:
| System | Sonnet-4.5 | Gemini-3-Flash | Prior Sonnet-3.5 |
| Gemini DeepResearch | 0.692 [.640, .741] | 0.810 [.751, .864] | 0.693 |
| OpenAI DeepResearch | 0.620 [.569, .670] | 0.771 [.723, .818] | 0.703 |
| Grok3 DeepSearch | 0.579 [.520, .637] | 0.618 [.569, .667] | 0.441 |
Aggregate rank is stable; the top-two confidence intervals overlap. A paired permutation test finds Gemini ahead under Sonnet-4.5 (p=0.003, Cohen's d=0.39) but not under Gemini-3-Flash (p=0.219). Judges correlate 0.54 to 0.82 on which questions are hard, yet agree on the full three-system ranking for only 15 of 65 questions (23.1%). Criterion-level κ is 0.53 (78.5% raw). Critical-analysis items disagree 27.9% of the time; enumeration and depth sit at 14-15%. Gemini-3-Flash is about 5-6 times cheaper and keeps the same ranking.
CHARM-100 is a new synthetic chatbot set with six mixed-type criteria. Under Gemini-3-Flash, factual accuracy hits 87.0% exact (κ=0.642). The four ordinal criteria land at 38-58% exact but 85-93% adjacent; naturalness has the highest quadratic-weighted κ at 0.719. Response length is 81% exact, with 0.70 recall for brevity and 0.14 for verbosity. Mean score bias is +0.170; score-level Spearman is 0.810.
A configuration sweep on 80 test items (Default = shuffle + 3-shot + SKIP) shows judge family dominating:
| Judge | Exact | Mean κ |
| Gemini-3-Flash | 60.4% | 0.679 |
| GPT-5.4-nano | 51.0% | 0.462 |
| LLaMA-3.1-8B | 19.4% | -0.001 |
Dropping few-shot or shuffling moves the three families in different directions. Same-model ensembles add at most 1.1 points for Gemini and cost GPT 2.0-4.1 points. A cross-family panel scores 57.1%/0.626, worse than Gemini alone. There is no universally helpful mitigation stack.
Two downstream uses. Llama 3.1 8B reviews 10 papers against a 10-criterion binary rubric: 0.17 with no skill, 0.47 with a one-line skill, 0.82 with an expert skill. One rubric-guided rewrite reaches 0.85 and 0.86, with bootstrap intervals overlapping the expert. A GPT-5.4-mini regrade still shows 0.58 to 0.74 from vague to revised, below the expert 0.78.
For RL, Qwen3-4B-Instruct is LoRA-trained on 402 AdvancedIF prompts, with Gemini-3-Flash Autorubric scores as reward, about $12 for 50 steps. Validation rises from 0.756 to 0.795 at the best of ten monitored checkpoints (step 25; nominal Wilcoxon p=0.032, unadjusted for selection). Perfect scores go from 21/81 to 30/81; mean length drops from 1,097 to 733 tokens. Regenerating and grading with GPT-5.4-mini yields only +0.023 (p=0.243). On IFEval, eight of nine instruction types transfer non-negatively (punctuation +4.5 points; McNemar p=0.28).
Anyone scoring non-verifiable tasks, or using a rubric as an RL reward, gets a shared substrate for criterion types, shuffling, abstention, calibration, ensembles, and reliability stats. Per-criterion numbers expose holes a holistic score hides: the same length criterion recalls brevity at 0.70 and verbosity at 0.14. Those numbers also feed skill rewrites and training rewards.
This is infrastructure, not a new judging algorithm. RiceChem still trails the fine-tuned baseline. The CHARM sweep is one run per cell. Judge family swamps mitigation toggles. LLaMA-3.1-8B sits at κ near zero; the framework cannot rescue a weak judge.
Use it when you need per-criterion diagnosis, a skill-revision loop, or rubric rewards. Do not read the 0-1 score as a certified true grade.
The authors list ten. Ordinal judges cluster at scale extremes, so exact accuracy looks worse than adjacent accuracy. Each CHARM configuration ran once, so gaps include sampling noise. Verbosity bias is not handled. Prompts and experiments are English-only. A cheap judge can preserve a leaderboard and still miscalibrate at criterion level. Optimized skills couple to both rubric and model. The RL gain uses the same validation split for checkpoint picking and reporting. The second-judge RL check also resamples generations, so it cannot isolate judge coupling. The framework assumes a well-written rubric and does not certify construct validity.
CHARM-100 conversations and reference labels were model-written; a second pass on 50 stratified items yields macro-average κ=0.687. Synthetic control of the label joint is real; ecological validity is not. The RiceChem 2.7-point lift has no student-clustered inference. The RL p-value is unadjusted, and IFEval transfer is not significant. Experiments snapshot the March 2026 COLM submission; by camera-ready the library had gained skill-harness integration and rubric induction, and those runs were not repeated under v1.5.3.