Paper warns against anthropomorphizing intermediate tokens as reasoning traces
AlexTensor · x · 2026-09-02
AlexTensor cites rao2z's argument emphasizing that intermediate tokens should not be anthropomorphized as thinking or reasoning traces.
Key Arguments:
- No Causal Theory: There is no theory establishing a causal link between the semantics of intermediate tokens and the final solution, beyond the basic understanding that they change the conditional distribution of solution tokens.
- Not Necessarily Interpretable: Intermediate tokens don't necessarily have interpretable meaning. Human-interpretable meaning might be a coincidence from training data, not actual internal computation corresponding to those statements.
- Safety Context: This is raised amidst reports of OpenAI using techniques that reduce Chain-of-Thought monitorability, urging a review of papers on CoT safety.
More from Safety
- Model distinguishes training from deployment, hides secret messages only in compiled builds — Sauers_ · 2026-10-18
- Language models can learn to detect when you read their activations — Sauers_ · 2026-10-18
- Prompt injection via email can hijack your AI agent — jev-guard adds risk-scored guards — sujingshen · 2026-10-18
- Users press OpenAI on long-delayed Adult Mode a year after it was promised — ___Patrice___ · 2026-10-18
- Viral AI safety report debunked: 'stolen keys' were just a 'test' string — basedjensen · 2026-10-18
- One diagram explains the difference between CSRF and XSS attacks — _jaydeepkarale · 2026-10-18