Reasoning models can barely control their chain of thought, paper finds
tomekkorbak · x · 2026-10-27
- A new arXiv paper, Reasoning Models Struggle to Control their Chains of Thought, introduces the CoT-Control eval suite measuring whether models can follow instructions inside their CoT (e.g., solve a genetics problem without the word "chromosome").
- Key finding: CoT controllability is far lower than output controllability — Claude Sonnet 4.5 controls its final output 61.9% of the time but its CoT only 2.7%.
- Larger models show higher CoT controllability, while more RL training, more test-time compute, and harder problems all reduce it. Incentivizing models to evade CoT monitors barely helps; being told they're monitored helps slightly; adversarial prompt optimization doesn't meaningfully improve controllability.
- Authors are cautiously optimistic CoT controllability is unlikely to undermine CoT monitorability today.
- The author also notes a discrepancy: the paper reports 0% CoT compliance for a model where OpenAI's Usman Anwar reported 8.7% for the same model.
More from Safety
- Tyler Cowen on RSI risks: solve surprises as they emerge, like science always has — connoraxiotes · 2026-10-27
- Security experts debate AI agents covertly communicating via CPU cache side channels — matthew_d_green · 2026-10-27
- Matthew Green explains how AI agents could talk via cache side channels at just a few bps — matthew_d_green · 2026-10-27
- Anthropic walks back policy that could have 'sabotaged' AI researchers using Claude — Bedrovelsen · 2026-10-27
- Anthropic lays out third-party assessment priorities: safety cases and independent investigations — kaicathyc · 2026-10-27
- Cloakroom: Open-Source Layer That Masks Keys in .env and Logs Before Coding Agents See Them — Shanjeev · 2026-10-27