Robustness of IR Models to Collection Growth
Emmanouil Georgios Lionis, Debasis Ganguly, Sean MacAvaney
cs.IR, cs.CL
2026-08-25
Merging TREC-COVID into MS MARCO, Glasgow finds BM25 nDCG@10 falls from 0.645 to 0.503. Independent first-stage retrievers degrade less; rerankers barely move.
Live collections grow every day. A well-behaved ranker should not demote already-relevant documents just because unrelated pages arrived. The University of Glasgow writes that down as a Collection Growth axiom and tests it with a controlled merge.
They join two standard sets with almost no topical overlap: post-2019 TREC-COVID and pre-2019 MS MARCO / TREC DL-2019. The union has 9.1M documents; COVID is 1.9%. For a COVID query, MARCO passages are off-topic noise. For a DL-2019 query, COVID docs are almost all irrelevant. That lets them measure how much nDCG moves when noise is added.
Models are split by whether scoring looks at other documents. MDA (Multi-Document-Agnostic) scores an isolated query-document pair: dense RetroMAE, learned-sparse SPLADE, pointwise monoELECTRA. MDD (Multi-Document-Dependent) uses collection statistics, candidate lists, or neighborhoods: BM25 IDF, RM3 / VectorPRF, Contextual Document Embeddings (CDE), listwise Set-Encoder.
They report nDCG@10, first-stage quality (P@100 on COVID, R(rel=2)@100 on DL-2019), and Collection Precision@10, the share of the top ten that still comes from the subcollection the query belongs to. PRF uses the top 10. Rerankers rerank the top 100.
The drop is one-sided. COVID ΔnDCG@10 sits between about -0.244 and -0.019, so no model meets the axiom. DL-2019 barely moves, about -0.009 to +0.002, with CP@10 near 1. The authors point to MS MARCO pretraining and the 98.1% dominant subcollection.
At first stage, MDA holds up better. On heterogeneous TREC-COVID, RetroMAE reaches 0.735 nDCG@10 and SPLADE 0.700. BM25 falls from 0.645 to 0.503 (-22.0%); CDE from 0.801 to 0.716 (-10.7%). SPLADE only loses 3.9%, RetroMAE 4.8%.
| First stage | Hom nDCG@10 | Het nDCG@10 | Δ |
| BM25 | 0.645 | 0.503 | -22.0% |
| CDE | 0.801 | 0.716 | -10.7% |
| SPLADE | 0.728 | 0.700 | -3.9% |
| RetroMAE | 0.772 | 0.735 | -4.8% |
PRF does not rescue the small collection. On COVID, RetroMAE plus VectorPRF drops heterogeneous nDCG from 0.735 to 0.728; CDE's CP@10 falls from 0.924 to 0.906, so feedback steers the query toward MARCO. Rerankers are nearly immune: CDE then monoELECTRA is 0.779, Set-Encoder 0.775. On DL-2019, Set-Encoder is slightly higher (0.781 vs 0.767 behind SPLADE).
Production indexes are not frozen test collections. The paper turns "the corpus grew and the metric fell" into a measurable axiom, and locates the damage in first-stage retrieval of the minority subcollection. IDF-style models are especially sensitive to incoming off-topic mass. If stage one already scores documents independently, listwise rerankers do not add extra fragility to this kind of noise.
PRF deserves a separate note: it reinforces the already-dominant subcollection. Queries from the minority topic get extra interference from their own feedback loop.
The authors used one corpus pair, a 1.9% injection ratio, and neural retrievers pretrained on MS MARCO. How much of the asymmetry is inter-document dependence, and how much is pretraining domain plus size, cannot be separated here. The axiom allows a small ε; the paper never says which ε is acceptable. DL-2019 contributes only 50 queries on the large-collection side. A pairing with more overlap and more even sizes could shrink the MDA lead.