Rubric-Calibrated Preferences: Cross-Query Calibration of LLM Judgments via Item Response Theory
Fabian David Schmidt, Donato Crisostomi, Carlos Lassance, Nils Reimers
cs.IR
2026-09-29
RCP fuses listwise LLM prefs with a five-criterion rubric via IRT. Query-mean human correlation rises from 0.538 to 0.795; RCP-nDCG wins 72.4% of 185 split contests.
Rerankers decide which documents users and LLMs see first. The usual score, nDCG, sits on human relevance labels (qrels) that are expensive, sparse, noisy, and discrete. Once 14 rerankers sit close together, those labels stop separating them. On 45.6% of NanoBEIR queries and 30.7% of BRIGHT queries, nDCG saturates (everyone scores 1.0), floors (everyone scores 0), or compresses (half or more of the pairs tie).
An LLM judge can label every candidate. Comparisons inside one query split near-ties, but those scores have no shared unit across queries. Absolute grades share a unit and then tie documents that differ only a little. The gap is a score that keeps within-query resolution and still means the same thing from query to query.
Rubric-Calibrated Preferences (RCP) treats documents as examinees and a shared yes/no rubric as the test. Item Response Theory (IRT) then puts every examinee on one ability axis.
The primary judge is Qwen3.5-397B-NVFP4. GPT-OSS-120B repeats both stages on NanoBEIR, BRIGHT, and TREC-DL. TREC-DL’s main numbers use the smaller Qwen3.6-27B. Pools are RRF of Cohere Embed v4, Octen-Embedding-8B, and BM25, top 150, with missed positives swapped into the bottom of the pool. Evaluation covers 13 NanoBEIR datasets (649 queries), 12 BRIGHT tasks (1,366 queries), eight ViDoRe v3 corpora (2,419 questions), and TREC-DL 2019/2020 (43/54 queries), with 14 rerankers from six families.
Forty-six contractors graded 311 contests blind (2,268 documents, 7,080 grades). NIST’s TREC-DL grades share no annotators with that study.
| Check | Metric | RCP | Baseline |
| Query-mean vs human grade | Spearman ρ | 0.795 | raw BT 0.538 |
| Useful vs not-useful docs | AUC | 0.910 | qrels 0.651 |
| Exactly one metric matches humans | share of 185 contests | 72.4% | chance 53% |
| NanoBEIR, 91 reranker pairs | paired t-test separations | 63.5% | qrel-nDCG 34.1% |
Against the judge’s own 486,750 criterion answers, 2PL ECE is 0.017 versus 0.371 for a sigmoid on raw BT. ECE against human grades is 0.054. Pooled Spearman of score vs mean grade rises from 0.641-0.693 to 0.753-0.830, in range of a second annotator panel (0.773).
The leaderboard flips. On NanoBEIR qrel-nDCG@10, Qwen3-Reranker-8B and Contextual v2 6B lead at 74.7 and 74.5; the three ZeroEntropy models sit last (66.5-67.0). Under RCP-nDCG@10, zerank-2 and zerank-1 move to 87.2 and 87.1 at ranks 1 and 2; the old leader falls to rank 9. Zerank trains on an LLM-ensemble preference signal, so judge circularity is a live concern. Drop those contests and RCP still matches humans on 68.1% of splits. In the 66 zerank disagreements with a verdict, annotators pick zerank 78.8% of the time. On TREC-DL, RCP-nDCG backs NIST’s winner on every pair NIST itself separates, with either judge. When the metric margin is at least 0.20, RCP matches the human verdict 96.8% of the time against 53.1% for qrel-nDCG.
Count-nDCG, rubric only, separates about as many reranker pairs as RCP, so most extra discrimination comes from the criteria. Three blind LLM judges still prefer RCP’s ideal top-10 over Count-nDCG’s on 80.6% to 98.8% of queries with a preference. The tournament is what breaks rubric ties. Among graded top-five documents, annotators call 27.8% of qrel-negatives useful and 37.5% of qrel-positives not useful. Blind judges from other model families prefer the RCP-selected positive set on 77.9% of decided NanoBEIR queries and 87.8% on BRIGHT.
RCP is an evaluation method, not a new reranker. Label a pool once and score later models with dense gains. Code and judgments are public. If nDCG is tying your board, RCP-nDCG splits many of those ties in a direction closer to people. TREC-DL costs about $3.72 per query with Qwen3.6-27B. Half the judgments still give Kendall τ 0.987 against the full board and reverse no pair the full budget already separated.
A new stick reorders models that train on LLM preferences. Anything trained on RCP feedback should be tested on other labels. Using the same rubric to calibrate arena votes or as a continuous RL reward is named, not measured.
Gains and annotators score each document alone, so redundancy is free. On HotpotQA, a second copy of the primary fact can outrank the document that holds the secondary fact. Both sources measure perceived relevance, not factual correctness.
The annotators’ questions were written to echo the rubric. Document-level and contest-level human numbers share the same grades; they are not two independent studies. NIST grades do not share that pool. Two annotators land within one grade of each other 77.8% of the time; Krippendorff’s α is 0.571. When RCP-nDCG@5 differs by less than 0.02, agreement with humans is 47.2%, which is chance. Read those gaps as ties. The human contests also oversampled cases where the two metrics disagree.
Scores are comparable only inside one fit. A document outside the judged pool needs new windows and a refit that can move earlier scores. Stage A uncertainty is not passed into IRT. The rubric was tuned on 15 NanoBEIR queries with 20 documents each, about 0.3% of pooled documents. The human study drops NanoSCIDOCS and BRIGHT’s coding and theorem tasks. Cohere sells Rerank 4 Pro, Rerank 4 Fast, and Embed v4, all used in the experiments.