Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety
Tomek Korbak, Mikita Balesni, Elizabeth Barnes, Yoshua Bengio, Joe Benton, Joseph Bloom, Mark Chen, Alan Cooney, Allan Dafoe, Anca Dragan, Scott Emmons, Owain Evans, David Farhi, Ryan Greenblatt, Dan Hendrycks, Marius Hobbhahn, Evan Hubinger, Geoffrey Irving, Erik Jenner, Daniel Kokotajlo, Victoria Krakovna, Shane Legg, David Lindner, David Luan, Aleksander Mądry, Julian Michael, Neel Nanda, Dave Orr, Jakub Pachocki, Ethan Perez, Mary Phuong, Fabien Roger, Joshua Saxe, Buck Shlegeris, Martín Soto, Eric Steinberger, Jasmine Wang, Wojciech Zaremba, Bowen Baker, Rohin Shah, Vlad Mikulik
cs.AI, cs.LG, stat.ML
2025-07-16
A multi-lab memo argues CoT monitoring can catch misbehavior in reasoning models, is incomplete, and can be destroyed by ordinary choices such as process supervision.
Reasoning models write long thoughts in human language. That creates a safety channel that did not exist for ordinary chat models: read the chain of thought and look for intent to misbehave. Models have already said "Let's hack" while exploiting a reward function, and "I'm transferring money because the website instructed me to" under prompt injection. The trace is treated like an activation during outcome-based RL, not like a supervised utterance, so it can leak intent the output head was trained not to show. The same training stack can also erase that leak.
This is not an empirical paper. Authors from the UK AI Security Institute, Apollo, OpenAI, Anthropic, Google DeepMind, METR and others, with endorsements from Hinton, Sutskever, Bowman and Schulman, wrote a memo for frontier labs: CoT monitoring is worth investing in as one layer, it is fragile, and ordinary development choices can shut it.
The argument has two parts. Why current CoTs are often monitorable comes first. On hard tasks, long serial computation in a Transformer has to pass through generated tokens as working memory; information cannot loop from late layers back to early ones without the chain of thought. Pretraining also installs a strong natural-language prior, so those tokens tend to stay human-readable. The authors call this the externalized reasoning property. It says some reasoning is on the page. It does not say the incriminating part is on the page. Even when a model could skip CoT, it may still be disposed to think out loud. Disposition is not a robust control, but it can surface misbehavior that action-only monitors miss.
How that property dies comes next. Heavy outcome-based RL no longer pins language to human data, and older work already showed models drifting off English under that pressure. Process supervision that shortens, sanitizes, or safety-washes the trace can make the written reasoning diverge from the decision path. Preference models on the output, humans rating visible CoTs that later enter reward models, and using monitors to patch reward-hacking environments all put indirect pressure on the trace. Architectures that reason in a continuous latent space may not need to speak at all.
The research agenda is evaluations: readability, latent reasoning with CoT disabled, causal tests that perturb the trace, and end-to-end catch rates in agent environments. Developers are asked to put monitorability in system cards and to treat it as an input to train-or-deploy decisions, including rolling back to an earlier checkpoint. The paper does not take a side on showing CoTs to users, because display creates pressure to supervise the trace until it looks safe, which is one of the failure modes above.
There are no new precision, recall, or ablation numbers. The result is three consensus claims.
CoT monitoring is already doing work: catching reward hacks in training, reading deceptive alignment, spotting evals confounded by memorization or scaffolding bugs, and noticing when a model knows it is being tested. Those cases are cited from Baker et al. and from existing agent evals, not generated here.
It is incomplete. Like every other oversight method, it misses some misbehavior. Necessity on hard tasks is the more durable hook; once a model can plan a harm inside activations, a clean trace is not evidence of safety.
Labs can close it themselves. Direct CoT supervision, more outcome RL, and latent-reasoning architectures are all on current roadmaps. Monitorability should be tracked as a property, not collected as a free side effect of shipping a reasoning model.
For safety teams: a confession in the trace is not an alignment win. Treat CoT monitoring as one extra layer next to output filters and white-box probes, and assume the failure modes should not correlate. For trainers: length penalties, process rewards, and showing CoTs to raters can buy a tidy transcript and sell the monitoring channel. For product: hiding traces creates compliance pressure, showing them invites training the CoT to look harmless. The paper writes that tradeoff down and does not pretend there is a clean fix.
If a proprietary API returns the chain of thought to the caller, monitorability, confidentiality, and user privacy collide. This paper supplies the frame. It does not run that experiment.
The author list is long; the footnote says the text is personal, not institutional. This is a position paper. The success stories it cites come from early reasoning models and are not compared under one protocol. Whether the worst harms actually need long serial reasoning is left open; many dangerous actions, once a model is trusted with tools and privileges, do not need a lot of thought. Process supervision is treated as a two-way bet: it may wreck monitorability, and it may also be a direct way to steer reasoning toward safer policies. The authors explicitly warn against freezing today's CoT form at all costs. There is no threshold for "monitorable enough" and no released eval suite. What you leave with is a research agenda and a lab checklist, not a leaderboard.