JEV-as-a-Judge: Accept When Confident, Escalate When Unsure
Yubo Li, Yidi Miao, Ramayya Krishnan, Rema Padman
cs.AI
2026-09-22
CMU finds TypeSafe JEV within 1.3 points of GPT-6 on RewardBench at $0.044 per 1,000 calls; a frozen confidence cascade keeps 99% of GPT-6 accuracy.
LLM-as-a-judge scales preference, factuality, and correctness checks, but two constraints show up at volume: the inference bill, and whether a given verdict can be trusted. Reasoning models can spend extra tokens on hard cases; they also overthink easy ones. Verbalized confidence is often too high.
This Carnegie Mellon study asks a narrower question. Can a decision-only judge, one that returns labels and label probabilities and no rationale, serve as a cheap first pass and flag the items that still need a stronger generative judge. The system under test is TypeSafe JEV 1.13.0, a hosted service. The implementation is proprietary, so the comparison is of complete judge configurations, not matched architectures.
JEV takes structured inputs and natural-language instructions and returns Choice, Noul, or Score distributions. The main runs send one Choice question per request. Native confidence correlates with the maximum label probability q at Spearman 0.971, 0.999, and 0.948 on RewardBench, JudgeBench, and HaluEval, so every plot, error detector, and routing gate uses q.
Seventeen configurations sit in the panel: JEV; GPT-4.1 mini through GPT-6 Astra; GPT-OSS 120B and Qwen3.6/3.8 27B on Groq; Claude Sonnet 5; Gemini 3 Flash and 3.1 Pro; local Qwen3-32B and Qwen3.5-27B; PairRM; and Skywork-Reward-V2-Qwen3-8B. Generative judges get the same instructions and must emit a decision plus probabilities with no rationale. Invalid outputs count as errors.
Public tasks: 400 RewardBench pairs, the full 350-pair JudgeBench GPT-4o split, 240 evidence-grounded HaluEval items, and 150 existing replies for final-answer adjudication. A 642-item pilot was frozen, then split 40/60 by source question into a selection set (temperatures and routing thresholds only) and a test set. A 670-item extension was frozen before anyone looked at pilot accuracy. For preference cascades, each pair is judged in both orders and the aligned probabilities are averaged. Thresholds were locked on 96 selection pairs, maximizing coverage while staying within two points of the fallback.
Fees use collection-time list prices: JEV at $0.042 per million input tokens and free output. Latency comes from a separate 120-decision isolated panel, not the bulk quality runs.
On ordinary preference and evidence-grounded factuality, JEV sits next to the strongest judge. RewardBench: 92.2% versus GPT-6 at 93.5%, paired difference -1.25 points (95% cluster interval [-3.8, 1.5]). HaluEval: 87.5% versus 86.7%. Blind human adjudication of disagreements leans further toward GPT-6: 17 of 29 RewardBench disputes, a human-adjudicated gap of -3.0 points. Of 26 HaluEval items both judges "miss," 24 have labels the evidence does not support; after correction JEV is at 95.8% and GPT-6 at 98.3%. The claim that survives both label sets: JEV stays within three points of GPT-6 on these two workloads.
Hard correctness is a different story. On JudgeBench, JEV scores 78.6% against 93.1% for both GPT-5.6 and GPT-6 (-14.6 points). The hole is widest on reasoning (68.4% vs 95.9%, n=98) and coding (76.2% vs 97.6%, n=42). Adjudication sides with GPT-6 on 57 of 69 disputes and with JEV on one. On RM-Bench, JEV scores 84.0% when both answers share a style and 74.8% when the rejected answer is the more elaborate one; GPT-6 moves from 93.3% to 94.6%, a -19.8 point gap on the hard pairs. Reference-free natural prose is near chance for everyone: 52.5% JEV, 53.8% GPT-4.1 mini, 55.0% GPT-5.4, all still confident.
| Judge | RewardBench (400) | JudgeBench (350) | HaluEval (240) |
| JEV 1.13 | 92.2 | 78.6 | 87.5 |
| GPT-6 Astra | 93.5 | 93.1 | 86.7 |
| GPT-5.6 Sol | 93.2 | 93.1 | 89.2 |
| Gemini 3.1 Pro | 94.5 | 87.4 | 87.1 |
| Skywork-Reward-V2 | 94.0 | 71.1 | — |
On the isolated panel JEV's median latency is 0.152 s and the fee is $0.044 per 1,000 judgments. GPT-4.1 mini: 0.548 s and $0.390. GPT-6: 1.885 s and $12.182. That is about 0.36% of GPT-6's fee and about one twelfth the latency.
Confidence ranks JEV's errors. Accuracy is 47.7% on 65 items with q<0.6 and 99.1% on 322 items with q=1. At q≥0.9 the two judges are nearly interchangeable (95.8% vs 96.5%); on the items JEV would escalate, GPT-6 leads by 15 points. A post-hoc single-order cascade at τ=0.9 escalates 34% of items, scores 91.3% versus GPT-6's 91.7%, and spends about 47% of the fee. The pre-specified two-order policy is cleaner: against GPT-6 at τ=0.9, it accepts 53.7% of 510 extension pairs, scores 92.5% versus 93.1%, and uses 56.8% of the fallback fee.
This is an operating profile, not a new router. Use JEV first on ordinary preference, evidence-grounded factuality, and final-answer checks. Escalate when the job is checking a derivation or refusing a florid wrong answer. No tested judge grades reference-free prose above chance; confidence will not save that workload.
The deployment checklist is short: judge pairs in both orders, fit the threshold locally and re-check it held-out, count invalid outputs as errors, treat confidence as a routing signal rather than a certificate. Thresholds do not always transfer. The GPT-5.6 policy at τ=0.7 accepts 81% of pairs and drops 2.35 points, past the two-point tolerance.
One proprietary JEV version against a chosen set of configs, with unknown training overlap. The multi-family expansion followed inspection of the pilot, so those comparisons are exploratory. Human adjudication is one author on 183 disagreement items, not a random audit; human–LLM κ on RewardBench is 0.29. Cascade fees are offline simulations with no live sequential latency. On style-adversarial pairs, JEV is still wrong on about a third of items scored in [0.9, 0.95); a τ=0.9 cascade keeps only 96.5% of GPT-6's accuracy there.