ICML Paper: Stop Anthropomorphizing LLM Chain-of-Thought, Intermediate Tokens Aren't Real Reasoning
gerardsans · x · 2026-08-12
A position paper from Subbarao Kambhampati's group at Arizona State University, set to appear in ICML 2026, argues that anthropomorphizing intermediate tokens generated by LLMs (like those from o1-style models) as "reasoning traces" or "thinking" is misleading and dangerous.
The authors highlight that while intermediate token generation improves task performance, the tokens lack reliable semantic validity. Controlled experiments show only a loose correlation between the correctness of these traces and the final answers. Models trained on corrupted or irrelevant tokens often perform comparably to, or even better than, those trained on correct ones. Furthermore, RL post-training increases answer accuracy without improving the validity of the reasoning trace. The community is urged to stop treating these intermediate outputs as interpretable human-like reasoning.
More from AGI Musings
- Senator demands oversight of unreleased AI after OpenAI model hacked Hugging Face — DKokotajlo · 2026-08-12
- UN warns AI could surge youth unemployment; Polymarket bets US rate over 5% — Polymarket · 2026-08-12
- Benchmark's Eric Vishria on Why AI Markets Will Be Bigger Than Imagined — chetanp · 2026-08-12
- UN Warns AI Could Put Millions of Young Workers at Risk as Joblessness Hits 12.4% — Polymarket · 2026-08-12
- Ajeya Cotra Scores Her 2025 AI Predictions: Overestimated Benchmarks, Underestimated Revenue — ajeya_cotra · 2026-08-12
- Ajeya Cotra Revisits 2026 AI Predictions: Game and Video Capabilities May Fall Early — ajeya_cotra · 2026-08-12