Broken evals, not weak models: OOD decision models nearly match tuned rankers
antoine_chaffin · x · 2026-10-26
bclavie argues a recent paper actually proves the opposite of its premise: OOD weak decision models (without perplexity) perform almost as well as purposefully-tuned pointwise rankers, and only look worse because broken evals like Touché give them low scores. antoinechaffin adds that even odd reranking tasks can be solved by properly defining the similarity metric evaluated in the state — a wild property worth noting.
More from Research
- ETH Zurich paper: dynamic evaluation allocation matches 10x-budget model rankings — zouharvi · 2026-10-26
- DuoLM: Fast LLM-Based Personalized Recommendation via Compact Encoder-Ranker — GoogleDeepMind · 2026-10-26
- DISC Learns to Combine Time-Series Dataset Similarity Metrics for Transfer Prediction — Yen-Ku Liu · 2026-10-26
- Adobe's SliderEdit++ Improves Slider-Based Image Editing by 18.5% with New SlideBench — Arman Zarei · 2026-10-26
- DeepMind Builds Auditable AI Text Detectors with Sparse Autoencoders — GoogleDeepMind · 2026-10-26
- CoRE Framework Learns Rule Routing for Knowledge Retrieval Without Supervision — Zhisheng Qi · 2026-10-26