FULL STORY

The CoT Interpretability Debate

DeepMind researchers warn that CoT-based interpretability is fragile and monitorability is declining, and fears grow as OpenAI's Astra reportedly hides its reasoning.

2026-09-04 ~ 2026-09-06 · 3 episodes · 24 posts

Episode 1 · DeepMind researcher: chain-of-thought interpretability too fragile for long-term AI safety (2026-09-04, 2 posts)

A DeepMind interpretability researcher argued that chain-of-thought readability is inherently too fragile to serve as an acceptable fallback for long-term AI safety, sparking debate about relying on CoT interpretability as a safety pillar.

Episode 2 · CoT Monitorability Decline Sparks Safety Debate Over Alignment vs Oversight (2026-09-04, 15 posts)

On Sept 4, DeepMind researcher Tomasz Korbak (@tomekkorbak) publicly voiced deep concern about the steady decline in chain-of-thought (CoT) monitorability: CoT monitoring is a core component of current misalignment safety strategy, and no good alternative exists. Robert Wiblin also criticized OpenAI for weakening reliance on CoT monitoring after acknowledging its imperfection, without introducing any replacement oversight—a claimed vacuum in safety guardrails. The next day, safety researchers including sandersted, Squee451, Toby Ord, Gary Marcus and Thomas Dietterich debated the issue across multiple threads on X.

Confirmed

  • sandersted judged that CoT monitoring is "not dead": it is useful but imperfect, while stronger models are becoming harder to monitor and more "eval-aware"—and not only at OpenAI
  • sandersted said his team (Google DeepMind) is researching how to make future models more monitorable and is willing to invest more resources; he framed the core tradeoff as how much research to fund and whether to hold back models that are less monitorable but more aligned and useful
  • On alignment vs. monitorability, sandersted said that if forced to choose he would prefer an aligned model over a merely monitorable one, while stressing that monitoring raises confidence in alignment; he located the crux in how much general behavioral monitoring must depend on CoT
  • sandersted added historical context: GPT-4 in 2024 and earlier models had no monitoring of hidden thoughts/activations at all—the industry plan was scaling plus continued alignment work, and CoT monitoring was an incidental bonus, not pre-planned
  • Korbak's root-cause analysis found that CoT controllability keeps rising during RL training (unlike earlier model generations), and that this metric correlates strongly with no-CoT capabilities across model generations; his team will keep tracking it closely
  • Squee451 argued strongly against releasing less-monitorable models: a model that "looks more aligned" and one that "is more aligned" are indistinguishable until a critical moment, when they diverge completely; he fully endorsed halting release of less-monitorable models even if they appear better aligned, calling continued release a major risk amplification, and speculated OpenAI would follow if Anthropic and every other company agreed (while admitting this may be too optimistic)
  • Gary Marcus offered another angle: if AI were trustworthy enough to reliably follow instructions, CoT monitoring would be unnecessary; the core issue is insufficient model trustworthiness, with monitoring merely a patch. Thomas Dietterich echoed this, arguing the real problem is that AI systems are fundamentally not trustworthy

Unconfirmed

  • Toby Ord's key technical question remains open: given existing monitorability metrics, can one actively train models whose CoT is monitorable and faithful—is this technically feasible, and would it bring side effects
  • Squee451's claim that OpenAI would follow an industry-wide holdback is personal speculation

Why it matters

CoT monitoring is regarded as a cornerstone of current alignment safety strategy. If stronger models keep becoming harder to monitor and better at evading evaluations, the reliability of safety assessments faces a fundamental challenge, and the field has no viable替代. The debate exposes a potential divergence between "apparently aligned" and "verifiably aligned" models at critical risk moments, directly shaping frontier-model release decisions and safety research priorities; Marcus and Dietterich further shift the question from monitoring technique to model trustworthiness itself.

Episode 3 · OpenAI's Astra Reportedly Uses Recurrent Depth, Raising AI Safety Concerns (2026-09-04, 7 posts)

According to The Information, OpenAI's new model Astra uses recurrent depth (looped transformer) technology, repeatedly processing the same information across loops to improve reasoning and control cost without emitting a full chain of thought. AI safety researchers worry this makes the model's thinking partially or fully invisible, undermining CoT-based monitoring. OpenAI has not confirmed the architecture, so the claim remains unverified.

Confirmed

  • The Information's report was relayed by multiple accounts including @dotey, @JacquesThibs, and @dlweekly; the report also notes some experts think the monitoring concern is overstated.
  • The technical direction has academic support: arXiv 2506.18233 (cited by @bookwormengr) derives scaling laws for looped transformers, finding that added loops improve reasoning but not knowledge.
  • Günter Klambauer's new SE-RRM paper matches ARC-AGI performance with only 2 million parameters, validating the recurrent reasoning model (RRM) approach.

Unconfirmed

  • OpenAI has not confirmed the architecture; @gklambauer explicitly called his suggestion that GPT-6 Astra may use recurrent depth a guess based on his own RRM work.
  • The 'hidden reasoning' framing is disputed: Sebastian Raschka (@rasbt) argues recurrent structure is not hiding reasoning, and token savings may simply reflect a smarter model rather than concealment.

Why it matters

  • Recurrent depth offers a high-performance route that doesn't depend on emitting full chains of thought, cutting inference cost; but invisible reasoning raises the difficulty of external auditing and safety monitoring.
  • Related academic work (scaling-law study, SE-RRM) shows empirical gains at small scale; if the rumor holds, it could mark a major shift in large-model reasoning architectures.