Paper: retrieval recall ceilings LLM recommendation reranking — oracle eval inflates NDCG up to 95%, none beat CF
_reachsumit · x · 2026-09-24
A single-author paper exposes evaluation pitfalls and the "recall ceiling" effect in LLM-based recommendation reranking.
- Key finding: LLM rerankers evaluated under oracle protocols (ground-truth item guaranteed in the scored set) overestimate realistic NDCG@10 by 92–95% across three Amazon datasets.
- Recall ceiling: realistic retrieval covers only 2–19% of relevant items at K=100 across eight datasets, deterministically bounding any closed-candidate reranker's top-k NDCG (under leave-one-out, E[NDCG@k] ≤ Recall@|candidate window|).
- Results: under realistic retrieval, none of the tested strategies — prompt engineering, 168× model scaling, sequential models, supervised neural rerankers, LoRA fine-tuning, hybrid retrieval, score-aware prompting, LLM+CF fusion — significantly beat a simple collaborative-filtering baseline; giving the LLM upstream CF scores mainly makes it reproduce the CF order.
- Proposal: the Recall-Aware Evaluation Protocol (RAEP) — classify the retrieval-recall regime first, then evaluate reranking where the ceiling permits differentiation; in low-recall regimes, improving retrieval matters more than improving reranking.
More from Research
- Anthropic's enzyme research uses AlphaFold, showing LLMs and specialized models complement each other — JMateosGarcia · 2026-09-24
- Philosophers push on AI distinctions, citing new Lederman & Goldstein paper — rgblong · 2026-09-24
- Claude teams up with AlphaFold for novel enzyme research, showing LLM-specialized model synergy — JMateosGarcia · 2026-09-24
- ECCV 2026 paper: x0-prediction fixes inefficient diffusion in reconstruction-tuned RAE latent spaces — serrjoa · 2026-09-24
- RecCAR closes reciprocal cross-attention gap in joint video diffusion models — barilan · 2026-09-24
- Berkeley's Do as I Do turns everyday human videos into dexterous robot hand training data — micoolcho · 2026-09-24