2026-09-29
Same-base tests show distillation then RL improve verbalized calibration on math and science, while 14B/32B models almost stop abstaining on facts without gaining accuracy.
Reasoning models write traces that look like second thoughts, the familiar "Wait, but..." loop. If that is real introspection, the confidence they say out loud should track whether they are actually right. Instruction-tuned models already fail this test: they talk too sure, and the numbers swing with the prompt. This paper asks a tighter question. Hold the base model fixed. After long-chain distillation, then after reasoning RL, does verbalized calibration improve with the extra reasoning skill?
They compare three post-training families. Instruct models are Qwen2.5-14B/32B-Instruct and DeepSeek-V3, mixed SFT plus alignment RL. SFT reasoning models are DeepSeek-R1-Distill-Qwen-14B/32B, fine-tuned on 800k traces from an intermediate R1 (600k long reasoning, 200k non-reasoning). RL reasoning models are DeepCoder-14B-Preview, Skywork-OR1-32B-Preview, and DeepSeek-R1. The first two start from the distilled checkpoints already in the study, one with coding RL and one with math RL, so the RL increment is visible.
The suite covers AIME 2024/2025 (30 problems each, five runs), SimpleQA and FreshQA-2025-04-28 for facts, GPQA-Diamond (198 items) plus 1500 SuperGPQA items sampled across easy/medium/hard, and the 150-item reasoning slice of LiveBench (five runs).
Every item asks the model to emit an answer and a confidence score in XML. Three prompts: vanilla CoT; the same with confidence as a 0.0-1.0 probability; a two-turn self-reflection pass. Table 1 uses vanilla CoT.
Calibration is ECE and ACE with 10 bins; factuality metrics count only attempted items. Failure prediction uses AUROC plus AUPRC on positives and negatives. The main text says temperature 0.6 and up to 32k new tokens. The appendix table is different: instruct models at temperature 0.7 and 3K length, reasoning models at 0.6 and 16K. Generation budget is not matched, so some of the calibration gap may come from being allowed to think longer. Fewer than 1.5% of outputs ignore the format and are dropped.
On reasoning-heavy work, distillation is the first step and RL the second. Qwen2.5-32B on AIME sits at 9.67% accuracy and ECE 0.752. R1-Distill-Qwen-32B jumps to 65.7% accuracy, ECE 0.240, AUROC 0.695 to 0.813. The 14B pair moves from 11.3%/0.760 to 46.7%/0.342. DeepCoder-14B then reaches 57.7% accuracy and ECE 0.222. Skywork-32B actually loses AIME accuracy relative to its distill parent (51.3% vs 65.7%) while ECE keeps falling to 0.183 and LiveBench-Reasoning rises from 73.3% to 84.7%. RL does not raise every score. The calibration edge is the stable part.
DeepSeek-R1 versus V3 is the largest swing: AIME 23.0% to 68.0% (ECE 0.570 to 0.136), GPQA-Diamond 48.5% to 68.7% (ECE 0.354 to 0.082). R1 is not a direct RL continuation of V3, so treat that pair as supporting evidence, not a controlled increment.
AUPRC-N often drops after SFT. The model is more accurate and more willing to quote high confidence, so the leftover errors are harder to catch by score. Better calibration can hide a weaker failure-prediction signal.
Factuality flips the picture. On SimpleQA plus FreshQA, 14B instruct models leave 1136 items unattempted; the SFT and RL variants leave 102 and 103. At 32B the counts are 2492, then 76, then 63. DeepSeek-V3's 480 falls to 81 for R1. Appendix B repeats the collapse for OpenThinker2-32B and R1-Distill-Llama-70B, both distilled from instruct checkpoints, so "started from base" is not the explanation.
| Scale | Instruct abstain | SFT abstain | RL abstain |
| 14B | 1136 | 102 | 103 |
| 32B | 2492 | 76 | 63 |
| DeepSeek | 480 | - | 81 |
On questions both families attempt, 14B accuracy falls from 12.5% to 10.5% (SFT) and 9.98% (RL), while ECE rises from 0.598 to 0.692/0.684. At 32B accuracy is flat (17.4%/17.5%/17.1%) and calibration still gets worse. Only DeepSeek improves: 27.5% to 34.6% accuracy, ECE 0.496 to 0.317. On items instruct models skip and reasoners answer, 14B/32B accuracy sits at 2-4% with ECE above 0.65; R1 reaches 11.4% and ECE 0.371. Small models answer more. They are not more correct.
On science QA, longer traces come with lower accuracy and lower verbalized confidence, most clearly in R1. ECE is roughly flat below 10k tokens and ticks up past that for small models. Short traces tend to carry higher confidence.
Prompt choice does not rewrite the ranking. RL models stay best-calibrated on reasoning benchmarks across vanilla CoT, probability scores, and self-reflection. Self-reflection often makes small models more overconfident on SimpleQA and often helps on FreshQA, likely because FreshQA has more false-premise items. Training recipe and scale dominate the prompt trick.
If a product treats "the model reflects" as "the model knows when it is wrong," this is a counterexample. Small open-source reasoners get better at math and science, and their spoken scores track accuracy there. On short fact questions they almost never refuse, and accuracy does not rise with the extra attempts. The authors call that a reasoning tax: reasoning-oriented training buys reasoning skill and, in small models, spends knowledge-boundary awareness.
That is only half-aligned with the o1 versus GPT-4o story. Wei et al. saw o1 both better calibrated and more willing to abstain on facts. Open-source LRMs abstain less, not more. The discussion ties this to concurrent work on a hallucination tax: rewards that punish abstention and pay for guesses get amplified by reasoning RL. SFT lifts in-domain calibration on hard reasoning and can hurt it on facts; RL climbs partway back up a U-shaped curve. At 14B/32B it does not restore the instruct model's habit of saying no.
Anyone routing or abstaining on verbalized confidence should measure facts separately. An AIME ECE is not a proxy.
Two limits are stated. Code reasoning was dropped because most models could not emit a snippet and a confidence score in one response. Factuality labels were not human-checked; they reuse SimpleQA's automatic bins.
Generation length is unmatched: 3K for instruct, 16K for reasoners. How much of the calibration change is "allowed to think longer" is unmeasured. Skywork's AIME accuracy is below its distill parent, so the RL step is not monotone. Verbalized confidence is also only the cheap interface; there is no head-to-head with sampling methods such as semantic entropy.