LLM CoT often mismatches internal reasoning; new CIA metric and post-training fix it
eunsolc · x · 2026-10-09
A new arXiv paper by Yihuai Hong, Shauli Ravfogel, Chen Zhao and Eunsol Choi, "Making LLMs Say What They Think," tackles the gap between a model's chain-of-thought and its actual internal computation.
- Introduces CoT-Interpretability Alignment (CIA), a metric that uses interpretability tools to detect a model's internal reasoning strategy and checks whether the CoT tells the same story
- Evaluated on two-hop QA, hint intervention, and integer multiplication across three LLMs: alignment is limited at 44.8%–75.9%
- Using CIA (plus task accuracy and parametric faithfulness signals) as a reward for post-training substantially improves CoT faithfulness while maintaining or improving accuracy
- Ships with code, data, and analysis of generalization patterns, offering both an auditing framework and a path toward more trustworthy explicit reasoning
More from Research
- arXiv paper gives StopGrads theoretical grounding: convergence without an underlying loss — srush_nlp · 2026-10-23
- PEARL demo: an LLM agent that checks experiment results, revises plans, and re-tests — KevinKaichuang · 2026-10-23
- Berkeley's StyleStream 2.0 cuts streaming voice conversion latency to 520ms, 2x faster — Berkeley · 2026-10-23
- Math PhD student quits academia: AI is making math unrecognizable — pickover · 2026-10-23
- ICLR Submission Found Containing LLM's Own Meta-Comment Admitting Results Weren't Significant — _onionesque · 2026-10-23
- AKV Serving System Lets Agents Manage Their Own KV Cache, Cutting Cost 27.7% — _akhaliq · 2026-10-23