Why do retrieval eval biases hurt less than decision-model eval biases?
antoine_chaffin · x · 2026-11-19
Responding to the CovidRetrieval length-bias analysis, the author notes retrieval evaluations have plenty of biases too—e.g., many BEIR datasets favor early parts of the input—but wonders why they aren't as painful as for decision models. Possible reasons: the biases are harder to learn, or IR has far more diverse training data and evaluations. Or, the author admits, maybe IR is just as bad and it's wishful thinking. Endorses the call for more debiasing perspectives on evaluations.
More from Research
- Universal Data-Efficient Pretraining Claims 56x Gains on Molecules, 24x on DNA and Crystals — madhavsinghal_ · 2026-11-19
- Fine-tuned World Action Model Dream-1 Jumps 159%, Beats Frontier LLMs on Robot Tasks — paigeinsf · 2026-11-19
- Fine-tuned Dream-1 world action model jumps 159% on real-robot benchmark at 1/100th the cost — paigeinsf · 2026-11-19
- AI makes 'living papers' nearly free to build, but their durability is doubtful — _akpiper · 2026-11-19
- HiT image generator hits 1.48 FID learning hierarchy from pixels, fast reproduced — niloofar_mire · 2026-11-19
- DaisyRose-27B: RL on 144 H200s for 18 days lifts Terminal-Bench from 5.6% to 12.0% — ying11231 · 2026-11-19