NeurIPS position paper: LLM reports quietly dropped calibration curves since GPT-4

burny_tech · x · 2026-11-10

A NeurIPS 2026 position paper observes that detailed calibration curves, once a fixture of GPT-4's technical report, have vanished from nearly every major LLM release since. The authors argue this is a mistake and call for embracing calibration across training, evaluation, and deployment.

The cited GPT-4 data shows the pre-trained model was well calibrated on an MMLU subset (ECE ≈ 0.007), while PPO post-training significantly hurt calibration (ECE ≈ 0.074).
Core argument: as LLMs enter high-stakes decisions, whether stated confidence tracks actual correctness matters as much as raw capability.

Original post →

NeurIPS position paper: LLM reports quietly dropped calibration curves since GPT-4

More from Models