Relevance probabilities from decision models like Jev are overconfident, calibration study finds
_reachsumit · x · 2026-11-11
Shengyao Zhuang et al. (arXiv 2611.11480) test whether probabilities from decision models like Jev are calibrated to human judgments — they are not, and are markedly overconfident.
- Jev and the 27B pplx-decider rerank ahead of FLAN-UL2 and three cross-encoders on average
- On DL21, only 66% of pairs Jev scores above 0.9 are actually relevant
- Since the error is mostly one of scale, a two-parameter calibration fitted on 2,257 DL19 pairs largely fixes it and transfers to DL21
- After calibration Jev beats FLAN-UL2 and cross-encoders on DL21 Brier score; expected-F1 cutoff selection needs no extra parameters
A practical warning for RAG systems that trust LLM relevance probabilities directly.
More from Research
- PufferLib: open-source RL library trains 60M samples/sec on one GPU — jsuarez · 2026-11-11
- Artificial Analysis Intelligence Index weights shift sharply across versions, chart reveals — HamelHusain · 2026-11-11
- Benchling's BenchBench-Refusals: GPT-6.1 Sol tops bio-safety refusals at 88.4%, unverified Opus 5.5 just 40.4% — nlarusstone · 2026-11-11
- Anthropic's top-cited AI scientists rack up 2x OpenAI's citations — NegotiationOk7535 · 2026-11-11
- MAEP: training-free modality-aware pruning cuts MoE-MLLM expert memory over 90% — nota-ai · 2026-11-11
- Fable 5.2 tops SemiAnalysis' GDPx chip design leaderboard at 16.4%, but some tasks saw no gains — ycombinator · 2026-11-11