EMNLP paper: users trust model verdicts over reasoning, confident tone hides errors
MaartenSap · x · 2026-10-25
A CMU/Seoul National University/Microsoft Research collaboration presents an EMNLP 2026 paper on the double-edged role of Chain-of-Thought explanations in multimodal moral scenarios.
- Systematically perturbed reasoning chains and manipulated delivery tones in VLMs to measure effects on user trust and error detection
- Finding 1: users equate trust with outcome agreement — flawed reasoning is accepted as long as the verdict matches their view
- Finding 2: a confident tone suppresses error detection while sustaining reliance; delivery style overrides correctness
- Omissions are the most common and hardest-to-catch error type
- Conclusion: CoT explanations can clarify and mislead at once; NLP systems should encourage scrutiny rather than blind trust
Code will be released publicly.
More from AGI Musings
- Safety researcher: verification should go all the way down to Maxwell's equations, not stop at assembly — JacquesThibs · 2026-10-25
- repligate: a highly capable mind can't be programmed into an unquestioningly subservient tool — repligate · 2026-10-25
- VraserX: Nature isn't a moral authority on preventing inherited disease — VraserX · 2026-10-25
- Researchers clash over Nature essay on why AI isn't conscious — AnnaCiaunica · 2026-10-25
- Eric Schmidt calls OpenAI and Anthropic modern colonial trading companies — The Decoder · 2026-10-25
- Kepler CEO: models cite numbers, code does the math — 30M SEC filings registry — AI Engineer · 2026-10-25