Study: Decision (System One) fine-tuning never beats plain reranker fine-tuning on Qwen backbones
_reachsumit · x · 2026-10-26
An arXiv study by Dinzinger et al. tests whether the decision (System One) fine-tuning behind 2026's calibrated-probability models makes better rerankers than plain reranker fine-tuning. Across six Qwen backbones (0.6B–27B) and ten TREC DL/BEIR collections, decision fine-tuning never wins: from 0.6B to 9B, reranker fine-tuning leads by 5–29 nDCG@10 points (including zero-shot), and at 27B the two tie. Up to 9B, decision tuning adds nothing over querying the untuned model, and its calibration advantage vanishes under a slightly different question wording. Takeaway: if you need a reranker, fine-tune a reranker.
More from Research
- HKU/ZJU/Fudan with Lightspeed launch HiSTEVE: hierarchical VLA agent playing Minecraft via native mouse and keyboard — tencent · 2026-10-26
- Sony researchers apply flow matching to four-stem stereo music source separation in FLOMSS — Shuyang Cui · 2026-10-26
- Zhejiang University and Tencent propose Easy-3Dfy: decoupling object generation and layout for single-image 3D scene construction — Ziang Zhang · 2026-10-26
- One week: 80-year-old cells reset to under 20, AI designing working proteins — rand_longevity · 2026-10-26
- Anthropic interpretability demo shows neurons combining into tiny pirate/wizard personas — Sauers_ · 2026-10-26
- 17-page HMM chapter PDF from Jurafsky & Martin's NLP textbook shared — burny_tech · 2026-10-26