Cohere Uses Item Response Theory to Calibrate LLM Judgments, Lifting Human Correlation to 0.795
_reachsumit · x · 2026-09-29
A Cohere team paper (Schmidt et al.) proposes Rubric-Calibrated Preferences (RCP) to fix a core problem in LLM-judged reranker evaluation: judgments are comparable within a query but share no scale across queries.
- nDCG relies on costly, sparse, discretely graded human labels and increasingly fails to separate near-quality rerankers; relative judgments rank within a query, absolute grades share a scale but are too coarse;
- RCP combines a listwise Bradley-Terry tournament per query with a rubric of increasingly stringent yes/no criteria, then applies Item Response Theory (IRT) to place all queries' tournament scores on one scale;
- Against blind grades from 46 external annotators, correlation between a query's mean score and mean human grade rises from 0.538 to 0.795; useful documents rank above non-useful ones with probability 0.910 (AUC).
More from Research
- DeepMind's AlphaProtein Novo designs enzymes nature never evolved, from drugs to plastic breakdown — hey_abusiddik · 2026-10-24
- K2 Horizon opens 13T pretraining tokens and post-training data pipelines, most detailed yet — ChengleiSi · 2026-10-24
- Coronary angiogram stenosis detection: annotation format mismatch broke YOLO cross-dataset — Future-Structure-296 · 2026-10-24
- ICLR reviewer despairs over reviewing 'slop' papers as submission quality craters — chhaviyadav_ · 2026-10-24
- Stanford's Potts points to Figure 8 of "Why Larger Models Learn More": 1B models overwrite old copies — ChrisGPotts · 2026-10-24
- Chris Potts on operationalizing learning-dynamics features for training prediction — ChrisGPotts · 2026-10-24