Incremental Pooled LLM Evaluation for Cost-Effective Retrieval Model Selection
Max Nelson, Hanoz Bhathena, Aviral Joshi, Saket Sharma
cs.IR, cs.CL
2026-09-02
JPMorgan pools top-k hits and judges unique pairs once with GPT-4.1; nDCG@10 rankings match humans at ρ 0.69–0.95, and a 62-config bake-off costs $800.
A production RAG stack needs a ranking of retrievers, not a one-shot human test collection. TREC-style human pooling is slow to rebuild when a new embedding model shows up. Judging each system independently with an LLM wastes money on documents that several systems already retrieved. JPMorgan’s setting is a financial news QA index that must keep absorbing new dense, sparse, and hybrid candidates.
Classical pooling has a second failure mode. Unjudged documents are treated as non-relevant, so a system that retrieves novel hits is penalized for being different.
Each candidate retrieves its top-k. The union of those documents is judged once by an LLM on a 0/1/2 graded scale. Standard IR metrics are then computed from the shared labels. When a new system arrives, only the documents it uniquely contributes are judged; previous labels are reused as-is.
The judge is pinned to GPT-4.1 (gpt-4.1-2025-04-14) at temperature 0, one turn, no few-shot examples. The prompt asks for a two-line thought and a JSON score. The public study uses 11 systems: dense and hybrid variants of five embedding families, plus BM25, each retrieving 100 documents.
P@k and DCG@k depend only on a system’s own top-k, so they cannot move when the pool grows. Recall, AP, and nDCG share a per-query denominator, so pairwise order on a single query is preserved. Macro-averages across queries are not guaranteed by the metric definitions, and that is the part the experiments check.
The four public sets are FiQA, TREC-COVID, Natural Questions (first 500 test queries), and the financial multimodal set FinRAGBench-V. Spearman ρ between pooled-LLM and human nDCG@10 rankings is 0.909 (FiQA), 0.691 (TREC-COVID), 0.945 (NQ), and 0.909 (FinRAG-V). Across 55 system pairs, nDCG@10 order agrees 78%–91% of the time. Of 27 nDCG@10 disagreements, 20 (74%) sit inside the gold qrels’ own bootstrap noise; 7 are real rank flips.
Pooling saves about 65%–67% of judgments versus independent labeling on the public sets. In production, a 300k-page news corpus, 303 queries, and 62 configurations produced 766,350 unique judgments against 3.77M independent ones: 79.6% reuse, a 4.9x cost cut. The pool grew over four weeks. GPT-4.1 annotation cost about $800. The chosen production system is hybrid-emb3-large-256 (MAP@100 0.461), essentially tied with the 768-d hybrid (0.459).
Claude Sonnet 4.6 agrees with GPT-4.1 on 84%–92% of binarized item labels, but after re-judging the full TREC-COVID pool the system ranking still has ρ=0.89 on nDCG@10 and 89% pairwise agreement.
| Dataset | nDCG@10 ρ | Judgment reuse |
| FiQA | 0.909 | 65% |
| TREC-COVID | 0.691 | 66% |
| NQ | 0.945 | 66% |
| FinRAG-V | 0.909 | 67% |
| Production, 62 configs | — | 79.6% |
For teams that bake off embedding models on a schedule, this turns “re-judge everything” into “judge the new hits.” The same pool later answered questions that were not in the original plan, such as whether dimensionality still matters under hybrid fusion, at zero extra LLM cost. Absolute nDCG is biased by the LLM’s broader notion of relevance. The offset is roughly uniform across systems, so ordinal selection is the intended use, not a reported absolute score.
Pairs separated by less than 0.001 nDCG are ties for both humans and the LLM. The primary rankings rest on one pinned GPT-4.1 checkpoint; Sonnet was fully re-run only on TREC-COVID. Smaller or domain-specialized judges are untested. Every system is a single-stage retriever; rerankers are out of scope. Benchmark-level stability under pool growth is empirical, not a theorem. Many production queries were LLM-generated, so they may not match live traffic.