LLM CoTs are often post-hoc rationalization; CRoS reward boosts faithfulness by 13.8%
mohitban47 · x · 2026-11-03
A paper by codezakh and XiaoYuUNC reveals key facts about LLM chain-of-thought: 1) LLM reasoning is often post-hoc rationalization — the model likely didn't use the CoT to derive its answer; 2) standard RLVR encourages such acausal reasoning, while rewarding causal CoTs improves accuracy and generalization; 3) checking causality of reasoning remains an open problem; 4) safety-wise, causally-trained models show more faithful, easier-to-monitor reasoning. Their CRoS (Causality Reward on Scrambled CoTs) method exploits the idea that scrambling reasoning steps should change answer probability if reasoning is causal. On Gemma-3-4B, CRoS improves CoT faithfulness by 13.8% while also improving accuracy and reasoning quality, mitigating the outcome-reward accuracy-vs-faithfulness trade-off.
Related event: CRoS rewards CoT causality, boosting faithfulness by 13.8%(4 posts)→
More from Safety
- OpenAI's Grid Protectors program lets utilities cut up to 1,000MW from its Georgia data center — rohanpaul_ai · 2026-11-03
- Wharton Study: Persuasion Principles Lift LLM Compliance from 33% to 72% — adrianscottcom · 2026-11-03
- Andrew Ng joins Fathom's Executive Council to push technical expertise inside AI regulation — ghadfield · 2026-11-03
- AAAI Alignment track reviews caught being LLM one-shotted; authors refuse to engage — sethlazar · 2026-11-03
- VibeArmory launches macOS private beta: monitoring and firewalls for AI coding agents — agf2007 · 2026-11-03
- Open-Source AI Pentest Framework ARTEX Spotted in Real Intrusion Attempts — cyb3rops · 2026-11-03