Penn & Yale Paper: Conformal Prediction Calibrations Diverge — ReCal Makes Them Reproducible
burkov · x · 2026-08-28
Conformal prediction turns a trained model's scores into prediction sets with a chosen error rate and is often treated as a reproducible final calibration step. But a paper from Penn and Yale shows that two people calibrating on independent held-out samples will almost always get different thresholds and thus different deployed classifiers — a real problem when models must be audited, cached, certified, or independently recalibrated, and it opens room to rerun calibration and cherry-pick favorable results.
Contributions:
- Formalizes how much agreement between independent calibrations is actually possible;
- Introduces ReCal, which makes two independently calibrated classifiers identical with a chosen high probability while retaining the usual coverage guarantee;
- Derives lower bounds showing the extra data and larger prediction sets this requires.
More from Research
- Researchers suggest AI conferences should run at a loss to subsidize youth — 3scorciav · 2026-08-28
- TTPO: Test-Time Policy Optimization for Label-Free Math Reasoning — Aozhe Wang · 2026-08-28
- Aphanta: Diagnosing Task-Aligned Image-Edited Intermediates — Fudan-University · 2026-08-28
- Paper by 40 Authors: CoT Monitoring is a Fragile Opportunity for AI Safety — CFGeek · 2026-08-28
- HydroGym: 60+ environments to train AI for fluid dynamics control — ricardovinuesa · 2026-08-28
- Miles now supports RL training for Qwen and GLM models with high-performance kernels — ying11231 · 2026-08-28