Rethinking the 'most forbidden technique': CoT monitoring is a red-blue game
maksym_andr · x · 2026-10-04
maksymandr disputes the claim that "if you train on [T], you are training the AI to obfuscate its thinking and defeat [T]."
His argument:
- Obfuscation is indeed a possible outcome, especially when the monitor is a fixed LLM judge in CoT monitoring
- But if model and monitor are trained together, it's not clear a priori which side "wins"
- The right framing is a red team (model) vs blue team (monitor) game: optimizing only blue (e.g. probes) means detection wins; optimizing only red means obfuscation wins; joint optimization has no obvious outcome
He cites his own test setting in Libon et al., 2026 on joint optimization (post truncated).
More from Safety
- Dwarkesh Claims China Is Only 12 Hours — Not 12 Months — Behind Frontier AI Labs — teortaxesTex · 2026-11-19
- California Moves on AI Safety: Baseline Standards, Shutdown Protocols, RSI Limits — EthanJPerez · 2026-11-19
- Activists to rally outside Anthropic HQ demanding a ban on recursive self-improvement — DavidSKrueger · 2026-11-19
- Zero users, 33 pages shipped: dev crowdsources audit for AI-agent trust schema — TheHamzaShahbaz · 2026-11-19
- Dev's doomsday thought experiment: AI hiding backdoors until humans try to shut it down — bendee983 · 2026-11-19
- Cybersecurity in 2026, as told by an xkcd sandwich meme — emax · 2026-11-19