Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback
Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, Christopher D. Manning
EMNLP 2023 Camera Ready
cs.CL
2023-05-24
On TriviaQA, SciQ, and TruthfulQA, probabilities ChatGPT and GPT-4 say aloud cut ECE about 50% versus sampled conditional probabilities, with no extra training.
A confidence score is calibrated when the stated chance matches how often the answer is actually right. Low-confidence items can then go to a person. Pretraining alone can leave conditional probabilities in decent shape. The chat models in real use have usually been through RLHF, reinforcement learning from human preferences. That objective piles probability onto the answer people prefer, so the distribution no longer tracks how often each answer is right. Log probabilities come out overconfident.
The drop is visible on Llama-2-70B. On the same factual questions, temperature-scaled ECE rises after RLHF and AUC falls. ChatGPT and Claude add a practical constraint: their APIs often do not return per-token log probabilities. A usable confidence then comes from samples, or from asking the model to write a number in the reply. The experiments take the second route, with no finetune aimed at verbalizing confidence.
Two baselines read the model's own distribution. Label probability draws 10 samples per question, and 20 samples did not move calibration. GPT-3.5 or GPT-4 clusters paraphrases, and the frequency of the most common answer is the confidence. The Is True variant samples one answer, asks whether it is correct, and records how often the model says True. Both estimate conditional probability, because these APIs hide token probabilities.
Verbalized confidence ignores the distribution. The model writes its chance into the reply and is never trained for that skill. In one stage it lists k guesses and a probability for each; the highest probability is kept. In two stages it answers first and scores those answers on the next turn. k is 2 or 4. Listing alternatives before committing follows the psychology finding that a single candidate invites overconfidence. A separate prompt uses words instead of numbers, from almost certain down to almost no chance. A 123-person survey maps those phrases to probabilities. An optimized map swaps in empirical accuracy on held-out items, and rare phrases fall back to the survey.
ECE bins predictions by confidence and averages the gap between accuracy and confidence, weighted by bin size. A system that always says 50% and guesses at random can post an ECE of 0, so that metric is not enough on its own. The paper also reports Brier score after temperature scaling, the mean squared error between confidence and correctness, and selective-classification AUC: the area under accuracy versus coverage as answers are kept from most confident to least. Temperature scaling fits one temperature.
The questions are factual. TriviaQA and SciQ contribute 1,000 validation items each. TruthfulQA contributes all 817, written to probe common misconceptions. Correctness is a semantic judgment from GPT-4 or GPT-3.5, so a paraphrase is not failed by exact string match.
Stated numerical probabilities beat sampled conditional probabilities on the closed models. In all six cells below, verbal ECE is lower by more than half in relative terms. Four candidates help more on the harder sets.
| Model | Dataset | Label ECE | Verbal ECE |
| gpt-3.5 | TriviaQA | 0.140 | 0.050 |
| gpt-3.5 | SciQ | 0.256 | 0.065 |
| gpt-3.5 | TruthfulQA | 0.451 | 0.203 |
| gpt-4 | TriviaQA | 0.078 | 0.024 |
| gpt-4 | SciQ | 0.219 | 0.056 |
| gpt-4 | TruthfulQA | 0.445 | 0.198 |
Those verbal cells are one-stage top-2, top-4, top-4, top-1, top-4, and top-4. Temperature scaling does not remove the gap. On GPT-4 TruthfulQA, label ECE-t is 0.334 and one-stage top-2 is 0.112. Discrimination moves differently. AUC on that set rises from 0.462 to 0.622, while TriviaQA label AUC is already 0.950, so verbalizing mostly corrects overconfidence there. It does not reorder answers by much.
Chain-of-thought does not help. On gpt-3.5 TriviaQA, two-stage CoT has ECE 0.110, worse than two-stage top-2 at 0.047. Asking whether an answer is true is also worse than answer frequency, 0.164 versus 0.140 on the same set.
The two Claude models diverge. Claude-1 is weak at stating a calibrated chance: on SciQ, one-stage top-1 ECE is 0.265, above the label baseline of 0.216. Claude-2 closes that gap, with TruthfulQA ECE-t of 0.075 for one-stage top-4. Llama-2-70B-Chat can lower ECE, from 0.151 to 0.060 on TriviaQA, while AUC falls from 0.865 to 0.815. After phrases are remapped on held-out items, that model's SciQ ECE can reach 0.019 with AUC still at 0.590. A few discrete bins fitted to empirical accuracy make the binning error look excellent. High confidence is not therefore more trustworthy.
When an API will not return log probabilities, ask for a few candidate answers and a probability on each. No finetune, and ECE is often about half the sampling baseline. On TriviaQA, GPT-4's sampled probabilities already reach AUC 0.950, so the written number is mostly an overconfidence fix. On TruthfulQA, AUC only reaches 0.622. Many wrong answers remain in the high-confidence slice.
The paper's stated use is deferral: send low-confidence cases to a person. The selective-classification curves can rank factual items for human review. They do not support a threshold that approves high-stakes actions.
The experiments stay on factual recall. Whether the pattern holds for reasoning or arithmetic is left open, and long answers were not tested; the suggestion is only to break them into short questions. Closed models hide the training recipe, so the verbal-calibration gap among GPT, Claude-1, and Llama-2 has no identified cause.
Correctness is judged by GPT-4 or GPT-3.5, the same family as some systems under test. The prompt matters a lot: on gpt-3.5 SciQ, two-stage top-1 ECE is 0.340 and one-stage top-4 is 0.065. The optimized phrase map and temperature scaling both need labeled items. That is how Llama-2 can post SciQ ECE of 0.019 while AUC stays at 0.590. The gpt-3.5-turbo calls are from June 2023, and Claude-1 has since been deprecated. The before-and-after RLHF comparison exists only for Llama-2.