Paper: Fine-tuned Rerankers Beat Decision Models by 5–29 nDCG Points on Every Backbone Up to 9B
omarsar0 · x · 2026-10-28
A University of Passau paper, System One Models Are Not Better Rerankers, puts the hype around decision models (Jev, Key, Clef) to a controlled test as rerankers.
- Setup: same Qwen backbone (0.6B–27B, six sizes), same BM25 top-100 candidates, ten collections (5 TREC DL 2019–2023 + 5 BEIR); three variants per backbone: untuned zero-shot, decision fine-tune, and plain reranker fine-tune.
- Findings: from 0.6B to 9B, reranker fine-tuning beats decision fine-tuning by 5–29 points nDCG@10 on every backbone; at 27B they tie, so decision fine-tuning adds nothing over zero-shot prompting. The claimed calibration edge depends on the question phrasing and vanishes with slightly different prompts.
- Takeaway: "If you need a reranker, fine-tune a reranker" — and benchmark decision models on your own queries before switching.
Related event: Study: Decision Models Fall Short of Fine-Tuned Rerankers(4 posts)→
More from Research
- Abstraction neurons as embeddings beat SOTA on word similarity, fail badly on sentence retrieval — mayfer · 2026-10-28
- Purdue's IDEAS Lab demos GPS-free drone that tracks runners through dense forests on-chip — DavidSKrueger · 2026-10-28
- Webinar to present empirical research agenda for studying inter-AI morality — dioscuri · 2026-10-28
- T-DIG diversity policies zero-shot transfer to GPT Image 2 and Gemini — simi_97k · 2026-10-28
- Cohere Labs hosts Oct 28 research session with Lila Sciences and Physical Intelligence scientists — tw_killian · 2026-10-28
- Reward hacking in T-DIG: diversity-only reward yields blank images and off-prompt sets — simi_97k · 2026-10-28