Variations in Relevance Judgments and the Shelf Life of Test Collections
Andrew Parry, Maik Fröbe, Harrisen Scells, Ferdinand Schlatt, Guglielmo Faggioli, Saber Zerhoudi, Sean MacAvaney, Eugene Yang
cs.IR
2025-02-28
Re-annotating 43 TREC DL 2019 topics shows near-chance four-grade agreement, but system order holds at Kendall τ 0.879. Human nDCG@10 tops out near 0.81; RankGPT reaches 0.80.
Cranfield evaluations rest on an old result: assessors often disagree on a single document, and system rankings barely move. On TREC-4 and TREC-6, Voorhees measured judgment overlap around 0.4 and a Kendall τ of 0.890. TREC Deep Learning 2019 is looser. Passage queries have no description and no narrative. The corpus is MS MARCO passages, about 8.8 million short segments. Relevance has four grades, and the headline metric is nDCG@10. Four grades split more easily than a yes-or-no label.
The same leaderboard is reused as a training signal. ECIR 2024 and SIGIR 2024 alone cite it 32 times. Hard negatives are mined from high-scoring rankers, and distillation copies those rankers as teachers. Craswell and colleagues warned against iterating on the Deep Learning collections. This study re-judges all 43 DL'19 topics and asks whether rankings still hold, and whether the leaders have already met a human assessor.
Official judgments are the primary assessor. Eight retrieval researchers are secondary assessors, two per topic, in groups of 14, 12, 8, and 9 topics. Mean load is 1,125 pairs. The secondary pool keeps every document graded 1 or higher and adds 100 primary non-relevant documents per assessor. Agreement on non-relevant text is usually higher, so the thinner sample should barely move nDCG. Assessors saw the original grade definitions and knew the pairs had been judged. They did not see scores or the score distribution. Background lookup was limited to ChatNoir over ClueWeb22 with BM25, so a web search would not surface public DL'19 discussion.
Three label sets per topic give 3 to the power m possible collections. Samples are scored with nDCG@10. The chance that two systems swap order lies between 0 and 0.5. Figure 3 uses 10,000 combinations. Agreement with the official order is Kendall τ, Spearman ρ, and RBO. Score gaps use a paired t-test with Bonferroni correction. Rank changes use Wilcoxon.
Three further checks locate the disagreement. Grades are aggregated by max, min, and mean. Assessors wrote narratives, then re-judged low-, median-, and high-agreement topics against a fixed narrative, separating intent clashes from ordinary label noise. Each assessor's grade order is also treated as a TREC run and scored on the official qrels. Shuffling inside a grade moves the metrics by less than 0.0001 standard deviation.
The system list runs from lexical retrieval to LLM reranking: ColBERT, SPLADE ED++, monoT5 (base and 3B), distilled monoELECTRA, SetEncoder, RankZephyr, and RankGPT on GPT-4, GPT-4 Turbo, and GPT-4o. ColBERT and SPLADE are first-stage retrievers. The others rerank BM25 or ColBERT.
Four-grade agreement is near chance. In the four groups that include the primary assessor, Fleiss κ is 0.22, 0.28, 0.17, and 0.28. Binarized, those values rise to 0.44, 0.40, 0.31, and 0.37. Four-grade overlap is only 0.11 to 0.19, and Cohen's κ sits in the same band. Binary overlap returns to 0.45 to 0.48, in line with Voorhees's overlap of about 0.4. Between the two secondary assessors, four-grade overlap is 0.42 to 0.63 and binary overlap is 0.71 to 0.89.
Grades drift downward. Of 800 documents the primary assessor called non-relevant, 13% change grade, and 77% of those changes stop at grade 1. In a 10-topic pilot of the top 10 pooled passages, perfect relevance was 1.5% originally and 11.8% in the new labels. Highly relevant fell from 30.8% to 10.8%. The medical query 'types of dysarthria from cerebral palsy' has Fleiss κ of 0.06: one assessor read it as a relative asking about prognosis, the other as a clinical definition, and the second marked almost everything non-relevant. 'is cdg airport in main paris' has κ of 0.08; both assessors wanted distance or travel time. Nine documents moved from non-relevant to perfectly relevant.
Fixing the narrative raises Cohen's κ by 24% and 46% across the two assessor groups from the hard query to the median one, and overall agreement rises 79%. The median-to-easy step adds 1%. A written intent removes a large share of the clash. What remains is mostly how the short title is read.
System order still holds. Natural combinations of secondary labels correlate with the official ranking at Kendall τ 0.879, Spearman ρ 0.972, and RBO 0.888. Samples that also draw the primary labels reach τ 0.897, close to the 0.890 that Table 1 lists for Voorhees. Absolute nDCG@10 generally falls. Lexical systems barely move in rank. Smaller supervised and semi-supervised models often climb. Larger semi-supervised models move less. Listwise encoders tend to drop. GPT-4o reranking BM25 loses rank by a significant margin and stabilizes once ColBERT is the first stage. RankZephyr, trained to reduce listwise positional bias, often gains rank.
Scored on the official labels, human grade-orders reach a mean nDCG@10 of 0.81±0.05, with min aggregation at 0.76±0.07 and max aggregation at 0.79±0.05. Max takes the highest grade on each passage, so it can land under the mean. The 0.81 figure is the practical ceiling given here. On judged documents only:
| Method | nDCG@10 | P@10 | MRR@10 | R@100 |
| BM25 | 0.51±0.08 | 0.41±0.09 | 0.70±0.12 | 0.65±0.09 |
| SPLADE ED++ | 0.73±0.07 | 0.63±0.10 | 0.91±0.07 | 0.69±0.09 |
| ColBERT→RankZephyr | 0.77±0.07 | 0.70±0.09 | 0.85±0.08 | 0.67±0.09 |
| ColBERT→RankGPT-4o | 0.80±0.05 | 0.73±0.09 | 0.89±0.08 | 0.67±0.09 |
| Assessor mean | 0.81±0.05 | 0.71±0.10 | 0.90±0.08 | 0.86±0.06 |
RankGPT-4o's nDCG is not separable from the human aggregates. Its P@10 sits slightly above the human mean. Recall does not: human R@100 is 0.75 to 0.86, while these neural and LLM runs stay near 0.67. With unjudged documents left in, ColBERT→RankGPT-4o scores 0.78±0.06 and RankZephyr 0.75±0.07. No single system covered here is reported above 0.81. Large ensembles are excluded.
A τ of 0.879 means a second assessor pool does not overturn the board. The runs that move are LLM-distilled listwise rerankers on BM25. GPT-4o drops significantly on that weak first stage and recovers on ColBERT. RankZephyr, trained against that positional bias, tends to rise under the new labels.
The precision ceiling is already crowded. On judged documents, ColBERT→RankGPT-4o at 0.80 is 0.01 under the human mean of 0.81, and the intervals overlap. Another point on these labels is hard to read as better retrieval. Recall is not crowded: R@100 is 0.86 for the mean assessor and 0.67 for the leading rerankers. Chasing nDCG@10 is a poor fit when the task is coverage.
The training stack passes this taste along. Teachers and hard negatives chosen for a high DL'19 score teach the official qrel's interpretation. Smaller distilled models shift more under the new labels. Capacity is still tight even in domain.
Re-annotation is expensive. The paper does not offer a way to date a collection without a large judging budget.
The assessors are the authors. A footnote says the overlap has little effect. There is no outside assessor group. These are retrieval researchers, not the original NIST assessors. The pool comes from 75 submitted systems, 44% of them neural language models. Expertise, the annotation setup, and pool bias are mixed into the gap between old and new labels.
Only 100 non-relevant documents were sampled per assessor, against 200 in Voorhees. A 13% grade change means agreement on non-relevance is only partial. Per-model nDCG before and after the new labels is never tabulated. Claims of substantial degradation come from rank-shift plots, not a quoted delta.
Forty-three topics cannot support finer claims. Assessors outside the US found US-centric queries hard, and that bias is left open. The narrative gains of 24%, 46%, and 79% come from one low-, one median-, and one high-agreement slice. They are not a general estimate of what a narrative would buy.
Expiration is an indirect claim. Rank correlation stays high. What looks used up is the precision ceiling, plus the fit of some LLM systems to the official taste. A high citation count does not by itself show that later gains came from semi-supervision. Large ensembles were excluded, and those are the systems most likely to pass 0.81.