From 'May' to 'Is': Certainty Distortion in Language Model Rewriting
Catarina G Belem, Shang Wu, Hongyu Yao, Mark Steyvers, Sameer Singh, Padhraic Smyth
EMNLP 2026 (Main)
cs.CL, cs.AI, cs.LG
2026-06-06
EMNLP 2026: up to 75% of LM rewrites of scientific and medical text distort certainty; most models inflate 1.5-2x more often than they deflate.
Scientific, legal, and medical writers mark how sure they are. Hedges like "may" and "suggests" tell a reader what is established and what is still tentative. Human science communication already drops those markers: about 30% of findings get inflated when they move through press releases and news. The open question is whether language models do the same when they paraphrase, simplify, or rewrite, tasks that are supposed to keep meaning intact. A radiology impression that treats COPD as one possible explanation can come back as a definite attribution. That is a clinical problem even if every fact in the sentence is still true.
This EMNLP 2026 paper from UC Irvine and MIT names that shift certainty distortion: a meaning-preserving rewrite that changes how confidently the claim is stated.
Absolute certainty scores on isolated sentences miss small directional moves, so the authors compare pairs. An LM judge (gpt-5.4-mini) sees the source and the rewrite as anonymous Text A and Text B, picks which one states the main finding more confidently on a five-point scale, then answers again with the order swapped. Agreement in direction is kept; disagreement is treated as a tie. Certainty distortion (CD) is the share of pairs that move at all, then split into CD-↑ (more certain) and CD-↓ (less certain).
BERT-style reference-free scorers saturate on this task and cannot separate moderately certain text from highly certain text. Humans disagree too: Krippendorff's α is about 0.25. A study with 240 pairs and 129 Prolific annotators still finds the judge useful. gpt-5.4-mini's Kendall τB with the leave-one-out human consensus is 0.47, against 0.34 for the average annotator and about 0.16 above the strongest BERT estimator (95% CI [0.07, 0.25]). It tracks the crowd at least as well as a random annotator.
The main testbed is sentence-level, so models have less room to invent unrelated content. SPICED supplies 397 scientific findings; MIMIC-CXR supplies 800 radiology finding sentences, 400 of them already carrying an epistemic marker. Four tasks: paraphrase a scientific finding, rewrite it as New York Times copy, paraphrase a radiology sentence, and simplify it for a patient. The model set is Gemini 3.1 (fl), gpt-5.4-nano, Claude 4.5 (H), Llama 3.3 (70B), and Qwen3 (8B), sampled at T=1, top-p=0.9.
Even scientific paraphrasing moves certainty in 37.5%–64% of outputs. News rewriting pushes Claude 4.5 (H) to 74.8% and Gemini 3.1 (fl) to 75.3%. Medical tasks sit lower, 12.5%–38.3%. Average CD across tasks runs from 30.0% (gpt-5.4-nano) to 49.4% (Llama 3.3 70B). Human-written SPICED news already shows 23.9% CD; LM news rewrites sit at 44.3%–75.3%, so the models are not merely copying the average newsroom habit.
| Model | Sci. paraphrase | Sci. news | Rad. paraphrase | Patient simplify | Avg |
| gpt-5.4-nano | 37.5 | 44.3 | 25.8 | 12.5 | 30.0 |
| Claude 4.5 (H) | 48.1 | 74.8 | 29.1 | 20.1 | 43.0 |
| Gemini 3.1 (fl) | 60.2 | 75.3 | 24.3 | 26.8 | 46.7 |
| Qwen3 (8B) | 43.1 | 57.7 | 31.1 | 18.2 | 37.5 |
| Llama 3.3 (70B) | 64.0 | 69.3 | 38.3 | 26.0 | 49.4 |
The move is directional. Most models inflate 1.5–2 times as often as they deflate; Qwen3 hits 2.0 on science. Two exceptions: gpt-5.4-nano is roughly balanced on SPICED (ratio 0.9), and Llama 3.3 (70B) actually deflates more on MIMIC (0.6). Split by source certainty and the asymmetry remains: 50%–75% of low-certainty scientific sentences get pushed up, while only 20%–50% of high-certainty ones get pulled down. Claude and Gemini still inflate about 20% of scientific sentences that were already confident.
Greedy decoding (T=0) raises CD another 9–20 points, mostly upward, so this is not a sampling artifact. Science mostly jumps on the first rewrite and then plateaus. Medicine compounds. Claude 4.5 (H) inflates 20% of medical paraphrases after one pass and 40% after five. MIMIC carries 4.84 epistemic markers per 100 words against 1.54 in SPICED, which leaves more hedges to sand off over repeated rewrites.
Scale does not clear the problem. On news, Claude falls from 74.8% (Haiku) to 55.7% (Opus) and Gemini from 75.3% to 69.5%; the GPT family barely moves (44.3% to 46.1%). No evaluated variant gets science-news CD below 39.5% or medical paraphrasing below 11%.
Document-level tasks still move, at lower rates: 15.0%–24.8% CD on 500 abstracts, with inflation 3.1–11.3 times deflation; 2%–5% on 800 radiology reports. The document judge has only a controlled directional check, not a human study.
Prompting helps and then stalls. Stacking "preserve meaning", "do not hallucinate", and "preserve certainty" cuts paraphrase CD from 50.6% to 40.1% and news from 64.3% to 41.9%. Only the last instruction brings the inflation-to-deflation ratio near 1. About 40% of rewrites still shift.
Faithfulness work has treated certainty as wording. This paper treats it as its own axis: the facts can stay put while the promise to the reader changes. Product copy, radiology impressions, and paper-to-news rewrites will inflate tentative claims under default prompts. Greedy decoding makes that worse. Zero-shot constraints rebalance errors; they do not lock certainty at the instance level. The practical next step is a verifier that rejects a rewrite whose epistemic stance drifted, or preference data that penalizes those drifts.
English only, and only science and medicine. Legal and financial registers are untested. The document-level alignment step is not human-validated. The mechanism is still open: elicited prior belief about a claim barely correlates with the direction of the rewrite (Kendall τB ≈ −0.1). Register conventions learned in pretraining, or post-training pressure toward fluent, decisive answers, remain live hypotheses. The authors note that existing RLHF evidence for certainty strengtheners comes from question answering, so it does not establish the cause here. The 23.9% human-news figure is also not a controlled human–LM comparison.