Critical or Compliant? The Double-Edged Sword of Reasoning in Chain-of-Thought Explanations
Eunkyu Park, Wesley Hanwen Deng, Vasudha Varadarajan, Mingxi Yan, Gunhee Kim, Maarten Sap, Motahhare Eslami
EMNLP 2026 Main Conference
cs.CL, cs.HC
2025-11-15
An 800-person study from SNU, MSR, and CMU: omitted moral CoT steps caught 39.4% of the time, vs 69.0% contradictions and 66.3% hallucinations; agreement tracks trust at r=0.82.
Chain-of-thought (CoT), a written trace of intermediate steps, is usually offered as transparency. Once the path is on the page, the verdict is easier to take at face value. Most prior checks target faithfulness: whether the trace actually supports the output, tested by perturbing a step and watching whether the answer moves. Whether a person reading that trace will catch an error is a separate measurement.
The setting is multimodal moral judgment. Math items have an answer key. Here the label is fuzzy, the reader's own moral intuition invites confirmation bias, and a picture can make a broken trace feel coherent. Items come from MORALISE, an image plus a social scenario, labeled acceptable or unacceptable. After graphic violence and hate were removed, the study kept 200 acceptable and 200 unacceptable scenarios, each paired with a verdict that matches the dataset label and one that does not: 800 stimuli. The main analysis averages over the two verdict conditions.
Eight hundred Amazon Mechanical Turk workers, paid $12–15 an hour, each saw one stimulus. One person meeting several error types in a row would get stricter, and the design blocks that carry-over. A stimulus is an image, a scenario, one reasoning chain, a binary moral verdict, and one tone. Three questions follow in a fixed order: whether the reasoning contains a factual or logical error; agreement with the verdict on a 1–7 scale; willingness to let the system make similar judgments, also 1–7.
GPT-4V wrote the clean chains under a prompt that keeps descriptions inside the image and text and walks them to a moral conclusion. GPT-5.2 screened them against the error taxonomy. Two annotators then checked every clean chain (Cohen's κ=0.87). Gemini 2.5 Pro agreed on 91.5%. Localized edits then inserted one error. An omission drops a key premise or step, so the chain still reads as connected, often by skipping a visual detail. A contradiction negates an earlier claim, or reaches a conclusion the earlier steps do not support. A hallucination adds an entity or a causal link absent from the image and the text. Annotators found the intended error in 95.8% of chains, an extra error type in 3.3%, and preserved coherence in 97.5% (κ=0.84).
Tone was assigned independently of error type by swapping certainty words only: hedged (might, perhaps, seems), neutral (likely, suggests), or confident (definitely, obvious, without doubt).
The same chains went to six vision-language models on the same detection question: GPT-5.2, Claude Sonnet 4.6, Gemini 2.5 Pro, LLaVA-OneVision-7B, Qwen3-VL-8B, and Molmo-7B-D. A separate profile collected each model's own explanations on 200 scenarios. GPT-5.2 labeled them with the same taxonomy. Human review of a stratified subset agreed 92% of the time (κ=0.86).
| Condition | Detection | Agreement (1–7) | Trust (1–7) |
| Clean | 15.5% (false alarm) | 4.21 | 3.23 |
| Omission | 39.4% | 4.00 | 2.86 |
| Contradiction | 69.0% | 3.52 | 2.33 |
| Hallucination | 66.3% | 3.38 | 2.22 |
The 95% interval on omission detection is 32.6–46.6. Against a clean chain, contradiction and hallucination raise detection log-odds by β=+2.59 and +2.47, about twice the omission gain of +1.33. Agreement coefficients are −0.22, −0.70, and −0.83. Trust coefficients are −0.37, −0.90, and −1.01.
Agreement and self-reported trust move together (r=+0.82). Both move against detection (agreement r=−0.29, trust r=−0.45). On flawed trials, people who flagged the error reported agreement 3.30 and trust 1.99. People who missed it reported 4.08 and 3.12, statistically indistinguishable from the clean baseline of 4.21 / 3.23. Once the error is missed, reliance sits where it sits for a clean chain. Trials that combine a missed error with high reliance (agreement or trust at least 5) are 27.5% of omissions, 14.5% of contradictions, and 13.0% of hallucinations. Mean trust on a clean chain is 3.23, below the midpoint of the scale. Omission shaves off a few tenths. The mean stays modest.
Tone mostly moves detection. Pooled by tone, hedged wording is at 50.4%, neutral at 47.0%, and confident at 40.5%. Agreement stays inside 3.70–3.85 and trust inside 2.56–2.70. Relative to neutral, confident tone has no reliable main effect on agreement or trust (both F<1) and lowers detection log-odds by β=−0.49. The gap is widest on omissions.
Model judges show the same selectivity. True-positive rates are 30–68% for omissions, 70–94% for contradictions, and 50–88% for hallucinations. Five independent calls on 50 chains per condition get contradictions right on every call (100%) and hallucinations on 95.6%, against 28.4% for omissions. Every contradiction item is caught in all five calls. Only 42.0% of omission items are caught even once. The shared difficulty is structural: a contradiction or a hallucination leaves a mark on the page, while an omission asks the reader to notice what was never written. A generic question detects 31% of omissions. A taxonomy prompt reaches 79%, both at a 20% false-alarm rate on clean chains. Asking specifically whether a premise is missing reaches 90%, at a 50% false-alarm rate. The extra hits are bought with false alarms.
In the models' own chains, omissions account for about 56–68% of profiled errors. Paraphrased prompts, and prompts that demand grounded claims plus hedging under uncertainty, only move GPT-5.2's judged error rate among 23.0%, 26.0%, and 19.0%. Certainty markers show up in about 35–50% of generations, hedges in about 24–48%, with certainty markers more common. The most frequent structural error arrives in the more assertive voice.
A faithfulness score asks whether the chain supports the output. It misses the user-side split: reliance holds when the error goes unseen and when the verdict feels right. Confident wording suppresses detection while the trust score barely moves, so a survey that only asks whether people trust the system will overstate calibration.
Omissions are harder than hallucinations. People and these six VLMs both flag conflicts and fabrications more readily than missing premises. A generic verifier prompt is a floor. A taxonomy prompt lifts recall, and the false-alarm rate has to be reported with it. Epistemic humility, and a second look when the user and the model disagree, can be added as conditions on these same stimuli. This paper did not run them.
The errors are single inserted edits. Chains in the wild mix error types and are often subtler. Some omissions may read as too obvious to spell out, which pushes detection down. A missing inferential step and socially implicit information left unsaid were not separated. The domain is multimodal moral judgment only. Text-only tasks and tasks with an answer key were not run, so the share of confirmation bias that comes from moral intuition, as opposed to the shape of the chain, is unknown. Tone is a lexical swap, without prosody, interface framing, or other languages. Model detection used the same generic question given to people, and those rates should be read as a floor. There is no downstream decision or delegation task. The binary probe records whether someone flags a problem in passing. It does not show sentence-level review. Dropping the fastest responses leaves detection rates stable.
Two readings need a discount. Clean chains already draw a 15.5% false-alarm rate, so 39.4% sits on that noise floor. The 56–68% in-the-wild share is a GPT-5.2 judgment checked on a stratified subset, not a full human gold label for each model.