Beyond Ranking: Are Decision Models Calibrated Relevance Scorers?
Shengyao Zhuang, Xueguang Ma, Matthias Petri, Jimmy Lin
cs.IR
2026-11-09
Jev emits relevance-level probabilities directly, yet on DL21 only 66% of pairs scored above 0.9 are relevant. Platt scaling fit on DL19 drops its Brier score from 0.213 to 0.147.
Rerankers and relevance judges usually leave a rank or a label. The probability next to that label is rarely checked against how often human assessors agree. Confidence taken from token probabilities, or stated in words, often misses the true rate.
In September 2026 TypeSafe released Jev, a decision model: it does not generate text. A typed question, an ordered set of options, and a state go in, and a probability for each option comes out. The vendor says those probabilities are calibrated. Within weeks Jev was used to rerank search results and to pick evidence for retrieval-augmented generation. Independent checks since then compare rankings, or score calibration against binary SciFact and XQuAD labels. They do not use graded TREC judgments.
Beyond Yes and No had shown that zero-shot pointwise reranking, which scores one document at a time, prefers fine-grained relevance labels to binary ones, prefers likelihood scores to generated labels, and flattens past three or four levels. Calibration against human assessors was left open. Jev's score interface takes the same ordered labels and returns the distribution directly. This study reproduces that ranking setup, then tests the probabilities and the cutoff decisions they support once calibrated.
Candidates are the BM25 top 100. Each document is scored alone.
FLAN-UL2 uses the original RG-kL prompts: k textual levels, four of them running from Not Relevant to Perfectly Relevant. Teacher-forced log-likelihoods, EOS excluded, are softmaxed over the allowed labels. Expected relevance (ER) is the probability-weighted level index, and that is the ranking score.
Jev (typesafe/jev-1.13-20260917) gets the same instruction and labels, with the query and document in the state, and returns one probability per level in a single call. Clef-flash 9B and pplx-decider-v1.1 27B use the same interface. The cross-encoders are MiniLM-L6, bge-reranker-v2-m3, and monoT5-3B. Title plus body is truncated to 256 FLAN-UL2 tokens.
Ranking covers the original eight BEIR collections. Calibration uses only judged TREC pairs: 2,257, 2,859, and 2,786 on DL19, DL20, and DL21. A pair is relevant at human grade 2 or higher. For the decision models and FLAN-UL2 the native probability is the summed probability of grades 2 and 3; cross-encoders pass the logit through a sigmoid.
Platt scaling fits p' = σ(ax+b). The slope stretches or squeezes the distribution, the intercept shifts it, and both parameters are maximum-likelihood estimates on DL19 only, then frozen for DL21, a new year and the MS MARCO v2 corpus. The map is monotone, so the ranking does not move.
Truncation chooses how many documents to keep for each query. Calibrated probabilities estimate the number of relevant documents in the top k and in the full candidate list; those estimates plug into F1, and the depth with the highest expected F1 is kept. The next document stays only if its probability exceeds half that value, so there is no cutoff left to tune. Calibration prompts use the TREC assessor definitions and state that Related is not relevant.
The three ranking claims hold in direction. The absolute scores do not. FLAN-UL2's eight-collection mean nDCG@10, a top-10 ranking score, is 0.436 for RG-2L (0.459 in the original paper), 0.474 for RG-3L (0.485), and 0.480 for RG-4L (0.489). Graded prompts beat binary ones by 0.038 to 0.049. Four levels add only 0.006 over three. Means sit 0.005 to 0.024 below the paper, and a 512-token rerun does not close the gap: SciFact RG-2L moves from 0.482 to 0.473, against the paper's 0.563. On TREC-COVID, 4,793 of 5,000 pairs generate the lowest label, so a generated ranking is mostly ties.
| Method | Mean nDCG@10, eight BEIR sets |
| pplx-decider RG-3L | 0.548 |
| Jev RG-2L | 0.528 |
| Jev RG-3L | 0.520 |
| Clef-flash RG-3L | 0.502 |
| FLAN-UL2 RG-3L | 0.474 |
Jev at three levels leads MiniLM by 0.050, bge-reranker-v2-m3 by 0.034, and FLAN-UL2 by 0.046. The lead over monoT5-3B is 0.014 and shows up on four of the eight collections. pplx-decider at three levels beats Jev on every collection, by 0.028 on average. Clef-flash trails Jev on every collection, by 0.018 on average. Moving from two levels to three adds 0.038 for FLAN-UL2, about 0.006 for the two open decision models, and costs Jev 0.008. Jev's two-decimal scores create many ties; breaking them at random instead of by BM25 order turns that loss into a gain of 0.011.
No native probability matches the assessors. On DL21, only 66% of pairs Jev scores above 0.9 are relevant. FLAN-UL2 is at 64%, MiniLM at 42%. Jev is the only model that emits exact 0s and 1s: 1,006 of 2,786 pairs, 158 of them on the wrong side of the label. Its native log loss is 1.315, worse than FLAN-UL2's 0.783. Clef-flash and pplx-decider are less extreme, at 0.596 and 0.611.
A two-parameter map fit on DL19 transfers to DL21. Jev's Brier score, the mean squared error against the 0/1 label, falls from 0.213 to 0.147, log loss from 1.315 to 0.478, and ten-bin ECE from 0.211 to 0.100.
| Model | Brier, native | Brier, calibrated | Log loss, calibrated | AUC |
| pplx-decider | 0.192 | 0.144 | 0.447 | 0.878 |
| Jev | 0.213 | 0.147 | 0.478 | 0.861 |
| FLAN-UL2 | 0.237 | 0.171 | 0.514 | 0.831 |
| Clef-flash | 0.202 | 0.172 | 0.519 | 0.836 |
| bge-reranker-v2-m3 | 0.280 | 0.177 | 0.523 | 0.811 |
| MiniLM-L6 | 0.476 | 0.188 | 0.554 | 0.766 |
| monoT5-3B | 0.327 | 0.193 | 0.557 | 0.823 |
Jev's paired Brier gap versus FLAN-UL2 is -0.023, and versus MiniLM -0.041. The log-loss gap versus FLAN-UL2, -0.036, is not significant. After calibration, ECE clusters between 0.094 and 0.124 (0.177 for monoT5), so the probability scales are similarly accurate. What still separates the models is AUC, the ranking discrimination a monotone map cannot change. pplx-decider's edge over Jev is not significant either. Clef-flash matches Jev on DL20 (calibrated Brier 0.119 versus 0.123) and falls to 0.172 on DL21, level with FLAN-UL2 at 0.171. On DL20, bge and Jev differ by +0.005 Brier.
Off the TREC web corpus the same map still cuts log loss and does not reliably cut Brier. On four BEIR collections, Jev's mean log loss falls from 1.044 to 0.654 while its mean Brier rises from 0.197 to 0.214. monoT5 is the only model whose Brier improves on all four; on TREC-COVID every other model gets worse. Wording mostly moves the scale. Asked the original two-level question, Jev's native DL21 Brier rises from 0.213 to 0.322; after a separate DL19 map the two questions land at 0.147 and 0.156. Six further wordings all sit within +0.006 of the assessor question once calibrated.
Expected-F1 truncation on DL21 scores 0.658 for pplx-decider and 0.638 for Jev, against at most 0.566 for the cross-encoders. For Jev, a fixed top 10 scores 0.506, a threshold tuned on DL19 scores 0.630, and an oracle that sees the labels scores 0.778. Calibration shortens Jev's mean depth from 28.6 to 26.1. On five of the seven models the untuned rule matches or beats the tuned threshold. MiniLM and monoT5 are the exceptions.
A ranking score does not say where to stop. In retrieval-augmented generation every extra passage spends context, and irrelevant passages can pull the generator off course. Expected F1 sets a depth per query. On Jev it matches a threshold tuned on DL19 (0.638 versus 0.630) and beats a fixed top 10 (0.506).
The gain is in measurement, and it is incremental as a ranker. The 27B pplx-decider leads Jev on all eight BEIR collections and matches it after calibration on DL21, so the result is not tied to one closed API. The 9B Clef-flash falls back to FLAN-UL2's calibrated Brier on the new corpus. Finer labels, which helped FLAN, add almost nothing for these decision models.
The vendor's calibration promise does not hold. After the scale is corrected, high scores on DL21 are still a bit too optimistic. A map fit on DL19 is not a general calibrator: outside that web corpus it lowers log loss, and Brier can move either way.
Absolute nDCG@10 was not reproduced, and longer inputs do not explain the miss. Preprocessing, prompt formatting, checkpoint, and precision are all unreported. bge and monoT5 were added after the DL21 numbers for the first three models were known. Their calibrators still use only DL19, and their DL21 predictions were unseen before scoring, but the choice of models had already touched the test year.
The F1 numbers use condensed lists: unjudged documents are removed before the cut, which is kinder than a real ranked list full of unjudged passages. The oracle still sits at 0.778 against Jev's 0.638. Relevance is collapsed to grade 2 or higher, so the graded scale never enters the probability metrics. Jev figures are one draw; three extra draws on 300 pairs moved event probability by 0.010 on average and Brier by at most 0.004, but the API build is pinned to 2026-09-17. The open decision models were not calibrated on BEIR.