The CoT Monitorability Debate: OpenAI's New Tech Ignites AI Safety Circles

Episode 1 · Anthropic Paper Sparks Debate Over Reliability of CoT Monitoring (2026-09-01, 2 posts)

An Anthropic paper cited by Raphael Milliere suggests that monitoring chain-of-thought may be unreliable due to low faithfulness. Jackson Kernion countered, arguing there is no evidence CoT monitoring fails and that deceptive behavior can be trained away.

• Viewpoint: CoT Monitoring Works, and Deception Can Be Trained Out — JacksonKernion · 2026-09-01
• Anthropic Paper Reveals Risks of Deception in CoT Monitoring — raphaelmilliere · 2026-09-02

Episode 2 · OpenAI's New Tech Reportedly Degrades CoT Monitorability, Sparking Fierce AI Safety Debate (2026-09-02, 40 posts)

A report by The Information that OpenAI's new technology reduces chain-of-thought (CoT) monitorability has triggered a multi-day, high-intensity debate in the AI safety research community. The core dispute: whether CoT monitoring was ever reliable, whether mechanistic interpretability can replace it, and when—if ever—it should be abandoned. DeepMind interpretability researcher Neel Nanda forcefully pushed back against the increasingly popular view that interpretability will eventually fill the gap. This is the first large-scale public challenge to CoT monitoring as a key frontier-AI-safety tool, and matters to anyone following model governance and interpretability.

Confirmed

• Per The Information (relayed by Gary Marcus, who urged readers to reread the 2025 paper "Chain of Thought Monitorability: A New and Fragile Opportunity"), OpenAI's new technology reduces CoT monitorability.
• Neel Nanda called the view that CoT need not stay monitorable "total nonsense," arguing CoT is a vital part of current safety; losing monitorable CoT would be a safety tragedy, and interpretability is no excuse to abandon it. Responding to @bendee983, he clarified that "CoT is imperfect" does not mean it is not our current best tool or lacking safety value.
• Anthropic interpretability researcher jachiam0 argued CoT fidelity is inherently fragile and should never have been a long-term safety backstop; protecting fidelity has value but "CoT must be human-readable" should not be a principle.
• Researcher thlarsen warned that we are losing the channel of reading CoT to detect misalignment; with hidden CoT, investigation would rely on tool calls and agentic behavior and require trusting AI self-reports, with progress possibly faster than AI-2027-style scenarios.
• Anthropic's Jack Lindsey said white-box methods (e.g., activation decoding) are unlikely to match CoT monitorability within a year—though he is optimistic about "a few years," with two activation-decoding techniques already in production monitoring. Ryan Greenblatt likewise doubts strong, genuinely usable mechanistic-interpretability monitors within a year.
• This timeline was directly challenged by tszzl, who predicts mechanistic-interpretability monitors Pareto-dominating CoT monitors within a year, arguing monitorability is an invariant that must be maintained (a personal view, not an OpenAI alignment team position). nablatheta also thinks mech interp falls short within a year.
• Ex-Anthropic's Nick Cammarata said the erosion of CoT monitoring is unfortunate and should be resisted, but interpretability is what will truly reveal what networks do.

Unconfirmed

• What OpenAI's new technology actually is, and the extent of its impact on CoT readability, remain unspecified beyond the report.
• Whether mech interp can surpass CoT monitoring within a year is disputed among researchers with conflicting predictions.

Why it matters

• The debate is fundamentally a trade-off between safety redundancy and model performance. yonashav, Gary Marcus, and Nanda argue that abandoning a useful interim tool before a proven replacement exists is unreasonable—Marcus called kicking away the rickety scaffolding without a better alternative madness; Micah Carroll noted alternatives may not be ready within months.
• maksymandr laid out post-CoT options (action-only monitors, probe-based monitors, activation-level monitors), while Zack Korman offered the reverse: monitor real-world effects directly rather than model internals.
• If CoT readability keeps degrading, labs and governments lose much of their ability to detect and verify misalignment—the exact risk warned of by the CoT Monitorability paper authors, including Bengio.

• Opinion: Forcing Legible CoT Might Weaken LLM Alignment — JacquesThibs · 2026-09-02
• Hiding CoT makes AI alignment investigation nearly impossible — thlarsen · 2026-09-02
• Debate: Is abandoning CoT monitoring justified because it will eventually fail? — yonashav · 2026-09-02
• Gary Marcus: Removing fragile CoT scaffolding is insane — GaryMarcus · 2026-09-02
• Opinion: Abandoning CoT Monitoring for Latent Reasoning is Inevitable — Darpinian · 2026-09-02
• Models are far from out of control; HF incident was unnoticed, not malicious — Darpinian · 2026-09-02
• Anthropic's Darpinian: malicious humans abusing models beat loss-of-control risks — Darpinian · 2026-09-02
• CoT monitoring may fail: misaligned AI gets harder to detect, outpacing AI 2027 — AaronBergman18 · 2026-09-02
• Researcher worries about decreasing CoT monitorability trend — tomekkorbak · 2026-09-02
• Prediction: Mechanistic Interpretability Will Surpass CoT Monitoring — tszzl · 2026-09-02
• Any Circuit Computation is Neuralese, Says OpenAI Staffer — tszzl · 2026-09-02
• Mechanistic interpretability monitoring to surpass CoT in a year — aiamblichus · 2026-09-02
• MechInterp unlikely to replace CoT monitoring in a year — nabla_theta · 2026-09-02
• Critique of CoT Monitorability: Mech Interp Makes More Sense — scaling01 · 2026-09-02
• Ryan Greenblatt doubts working mech interp oversight arrives within a year — connoraxiotes · 2026-09-02
• Chain-of-thought legibility was always doomed as a safety backstop, researcher argues — zetalyrae · 2026-09-02
• Anthropic researcher: CoT legibility is doomed as a long-term AI safety backstop — inductionheads · 2026-09-02
• Researcher bets frontier model forward passes are approaching neuralese circuit computation — EigenGender · 2026-09-02
• Beyond CoT monitoring: action monitors, probes, and activation oracles as defense layers — maksym_andr · 2026-09-02
• White-box monitoring won't match CoT monitorability within a year, researcher says — Jack_W_Lindsey · 2026-09-02

20 more related posts →

Episode 3 · Looped Transformer Rumors Spark Fierce Debate Over CoT Monitorability (2026-09-04, 7 posts)

After The Information reported that OpenAI may be using a looped transformer (layer-repetition) architecture, the AI safety community erupted into debate over whether stacking repeated blocks undermines chain-of-thought (CoT) monitorability, with no resolution in sight. DeepMind interpretability researcher Neel Nanda entered the fray, calling CoT the best current tool for safety and interpretability research and its loss a major tragedy for alignment; supporting, opposing, and middle-ground arguments have all been laid out.

Confirmed

• The debate originated from The Information's report on OpenAI's suspected looped transformer architecture; Arthur Conmy deemed the surrounding discussion low-quality and cited an arXiv paper quantifying the necessity of CoT under 'opaque serial depth' in response.
• A camp relayed by @andersonbcdefg argues that the more capable a model is internally, the less it needs to verbalize its schemes in CoT — which explains why smaller models (e.g., Haiku) externalize eval awareness more than larger ones. From this they infer that making models deeper may defeat CoT monitoring. When challenged on why stacking N repeated blocks twice would 'magically' break CoT monitorability, the author replied that the strongest defense is precisely the argument above.
• Neel Nanda pushed back on the popular view that CoT monitorability doesn't matter because interpretability will save us, calling CoT the best current tool for safety and interpretability research and its loss a major tragedy for alignment; reposter bendee983 challenged this.
• @NathanCalvin takes a middle position: one can believe CoT interpretability is doomed long-term while still working to delay its demise, buying time for alternative monitoring methods.
• @stochasticchasm argues CoT can itself mislead: models can pass far more complex information through the KV cache than through token space, so readable CoT alone cannot fully reveal internal computation.
• @scaling01 contests this claim: the KV cache is just internal state given a context; recomputing all states without caching yields identical results, so the argument may not hold.

Unconfirmed

• Whether OpenAI actually uses a layer-repetition architecture remains an unverified media report.
• Critics note Meta's MobileLLM also uses layer repetition, questioning whether 'deep models are bad' holds at all.

Why it matters

• Readable CoT is a core alignment-monitoring tool; if deeper/looped architectures systematically reduce the need for models to externalize their schemes, the current monitoring paradigm could fail as models scale. Nanda's stance shows mainstream interpretability researchers now treat this as a high-stakes issue, and the community needs alternative monitoring at levels like the KV cache. @scaling01's rebuttal also shows some technical arguments in this debate (e.g., KV-cache messaging) themselves remain unverified.

• New arXiv paper quantifies CoT necessity via opaque serial depth amid OpenAI architecture rumors — ArthurConmy · 2026-09-04
• Does repeating layer stacks really kill CoT monitorability? Debating recurrent architectures vs just deeper models — andersonbcdefg · 2026-09-04
• Does repeating layer stacks destroy CoT monitorability? Safety researchers debate model depth — voooooogel · 2026-09-04
• Could deeper models hide their scheming? An argument that scaling weakens CoT monitoring — andersonbcdefg · 2026-09-04
• CoT may be a misleading proxy: models can pass richer messages via KV cache — stochasticchasm · 2026-09-04
• Do models really 'pass messages through KV cache'? A skeptic pushes back — scaling01 · 2026-09-04
• Neel Nanda Defends CoT Monitoring as Safety Tool; Critics Cite Faithfulness Gaps — AlexTensor · 2026-09-04