Kokotajlo challenges CoT monitorability evals: what if the model didn't refuse?
dfrsrchtwts · x · 2026-10-27
Former OpenAI researcher Daniel Kokotajlo raised a key critique of a paper on situational awareness and chain-of-thought (CoT) monitorability. The paper evaluated monitorability by (a) prompting the model to manipulate its CoT and (b) measuring capability loss with reasoning off — finding it far less capable without reasoning and that it typically refuses to manipulate its CoT. Kokotajlo's point: what if it didn't refuse? He wants evidence of the form "even if it tried its best to manipulate the CoT, it still can't" — possibly provable guarantees — and argues qualitative descriptions of model behavior would teach more than marginal eval numbers.
This is a substantive AI safety debate on CoT monitorability methodology.
More from Safety
- LLM bias judges name groups instead of saying 'unknown', refusing only 0.4% of the time — ZhijingJin · 2026-10-27
- ChatGPT will render genitals if you ask it to generate an X-ray of yourself — flowersslop · 2026-10-27
- Compaction Smuggling attack bypasses OpenAI mitigations to steal reasoning traces — teortaxesTex · 2026-10-27
- OpenAI backs California SB 53 safety law as AI governance shifts into practice — Comfortable_Gene5180 · 2026-10-27
- Rishi Bommasani: Anthropic's S-1 Shows the Value of Mandated Frontier AI Transparency — RishiBommasani · 2026-10-27
- Pasta Lab joins Artificial Analysis Cyber Index Alliance for AI security benchmarks — ArtificialAnlys · 2026-10-27