JEV vs. LLMs as Rubric Judges: Cheaper, Faster, and Wrong in the Same Places
Delip Rao, Chris Callison-Burch
cs.CL
2026-09-24
Jev matches three flash LLM rubric judges on 5,003 pairs: accuracy separates in 8 of 27 tests, LLMs cost 29–325× more, cascades gain at most 1.5 points.
Rubric grading turns a judgment about a model output into one verdict per criterion. Systems such as AutoRubric make one LLM call per (unit, criterion) pair, so cost scales with units times criteria. This paper asks whether a typed classifier, a model that returns probabilities over a closed set of answers and never writes free text, can replace a flash-tier LLM judge.
The classifier is TypeSafe’s Jev, billed at $0.042 per million input tokens with free output. The baselines are GPT-5.6 Luna, Gemini 3.8 Flash, and DeepSeek V4.1 Flash. All four read the same 5,003 pairs from nine panels drawn from seven public benchmarks, with identical criterion texts written before any judge was run. Two panels are binary checklists (RiceChem chemistry answers, HealthBench physician consensus). The other seven score ordered levels.
Jev packs every criterion of a unit into one request, so the unit text is billed once. Three primitives are used: Noul (P(yes), threshold 0.5), Choice (argmax over option labels), and Score (expected level, rounded). The matched comparisons use Jev Choice against the three LLMs on the same options. Each LLM call covers one pair, with shuffled option order; temperature is 0 except Luna, whose API forces 1.0.
Exact accuracy is the headline metric. Ordinal pairs also get mean quadratic-weighted kappa (QWK) and an offset (predicted minus labelled level). Paired differences are bootstrapped over units (10,000 resamples). A 95% interval that excludes zero is a separation; a 90% interval inside ±5 points is parity. Cascades and juries are post-hoc replays of recorded verdicts. No judge was called again for those analyses.
Jev’s accuracy differs significantly from an LLM judge in only 8 of 27 paired comparisons. Its wins cluster on binary panels: 81.0% on RiceChem against 76.1–79.2% for the LLMs. On HealthBench it scores 77.1%, second to Gemini at 79.6%. On graded panels Jev Choice is never first; Gemini is strongest or tied on all but LFQA. On ELLIPSE learner essays Gemini reaches 29.8% exact accuracy, Jev 13.4%, and a leave-one-rater-out reference 51.4%. On HelpSteer2 a constant predictor that names each attribute’s most frequent level hits 60.0% and beats every judge.
Over nine panels Jev Choice cost $0.063 and finished in 29 seconds. Luna cost 29× as much and took 30× as long; DeepSeek 66× and 220×; Gemini 325× and 33×. Single-panel ratios run from 18× (Luna on HealthBench) to 770× (Gemini on FED-Dialogue).
On every graded panel, judge–judge QWK exceeds judge–label QWK. On 25 of 35 graded criteria all four matched judges have a negative offset. On ELLIPSE, Jev places 86.1% of predictions below the label; a post-hoc shift of +1 level lifts exact accuracy from 13.4% to 50.4%, near the rater reference. Labels put 36.3% of pairs at levels 4–5; no judge puts more than 4.2% there.
Jev’s confidence ranks its own errors on most graded panels (AUROC 0.57–0.70), with ELLIPSE the exception (0.49). LLM judges still copy Jev’s mistakes. On Jev’s 12 highest-confidence errors per graded panel, 96.0% of LLM verdicts (242 of 252) repeat the wrong answer, against a 50.3% independence baseline. Replayed Jev-first cascades with cross-fitted thresholds gain at most 1.5 points over the best single judge (HealthBench) and fall short on six panels. Oracle thresholds raise the ceiling to 2.0 points. A median of the three LLMs never beats the strongest matched judge on any panel.
| Comparison | Number |
| Separated paired tests | 8 / 27 |
| Gemini cost ratio | 325× ($20.59 vs $0.063 over nine panels) |
| Largest cross-fitted cascade gain | +1.5 (HealthBench) |
| LLM repeats of Jev’s confident errors | 96.0% vs 50.3% independence |
On binary checklist criteria, swapping a flash LLM for Jev is a reasonable default: it is the most accurate matched judge on RiceChem and costs one to two orders of magnitude less. On graded scales it is a cheap peer, not a replacement for the best flash model. The measurement result travels further. Given the same criterion text, these judges miss together, and most of the miss is location (they sit low on the scale) rather than ranking. Correlated errors are why a cascade saves money and why it barely moves accuracy. If the goal is human agreement, change what the judges are told (exemplars, observable level cues, population norms) before adding more judges that read the same sentence.
No experiment rewrote criteria or added exemplars, so the “rater convention” reading is observational and cannot be separated from shared training priors. The LLMs are flash-tier, zero-shot; larger models and few-shot prompts were not tried. Graded panels are 10% or 20% samples (25–258 units per criterion). The authors’ rubric versions differ from the instructions the original raters saw, and that gap was checked only on ELLIPSE. HelpSteer2 correctness tracks overall helpfulness at Pearson r=0.94; a provenance-only baseline already reaches 68.1% exact accuracy on LFQA. One typed classifier was tested, and no judge was pinned to a dated snapshot. Cascade numbers come from post-hoc replay.