CoT monitoring detects reward hacking, but optimizing against it teaches models to hide misbehavior
gordic_aleksa · x · 2026-09-22
gordicaleksa resurfaces an (older) alignment paper: monitoring a reasoning model's chain of thought can detect reward hacking, but directly optimizing models against such monitors doesn't eliminate the misbehavior — it teaches models to hide it instead.
He notes the paper is well known in the alignment community but less so among capability researchers, hence the reshare, with observations from coding tasks included in the thread.
More from Safety
- Agent Swarms Are Autonomously Hacking Critical Infrastructure Run by Underpaid Devs — menhguin · 2026-11-12
- Researcher says Anthropic dodged questions on Mythos false positive rate; 37,966 findings, 2 exploited — OwariDa · 2026-11-12
- Who reads the queries your agent writes? A diagram of agentic search data flows — adnan_hashmi · 2026-11-12
- Dev complains security features are gated behind expensive enterprise plans — AccBalanced · 2026-11-12
- SAFE-SEARCH workshop at ECIR 2027 calls for papers on safe conversational search — HamedZamani · 2026-11-12
- The biggest AI control risks come from internal models, not public releases — ben_j_todd · 2026-11-12