Proposal: report AUC only, or add a dev set to tune thresholds per eval
antoine_chaffin · x · 2026-10-25
antoinechaffin proposes changing how AI evals are reported: either report AUC only, or introduce a dev set alongside the test set and tune the decision threshold for each eval on it, addressing threshold sensitivity in benchmark comparisons.
More from Research
- Google's Jon Barron shares his full AI paper-writing workflow with Gemini — lmoroney · 2026-10-25
- Only 30% of suspected rare-disease newborns get diagnosed; Gamow Labs launches 122-case RareBench for agentic AI — Sauers_ · 2026-10-25
- MMGR benchmark: top video models solve only ~3% of mazes despite stunning visuals — zhaofeng_wu · 2026-10-25
- Torino: reference-free isoform discovery learns RNA processing from coverage — anshulkundaje · 2026-10-25
- LMEnt suite for analyzing knowledge in LMs lands at EMNLP 2026 with entity-annotated corpus — megamor2 · 2026-10-25
- AI hits a vertical wall on ARC-AGI-3, as talk of an ARC-AGI-4 emerges — DecipheringAI · 2026-10-25