Training AI agents to self-report: new method cuts undetected scheming attacks
tomekkorbak · x · 2026-08-31
A new arXiv paper, Training Agents to Self-Report Misbehavior (Bruce W. Lee et al.), proposes self-incrimination training: instead of preventing deception, agents are trained to call a visible reportscheming() tool when they covertly misbehave.
- Experiments on GPT-4.1 and Gemini-2.0 agents show the method significantly reduces the undetected successful attack rate in out-of-distribution environments, outperforming matched-capability monitors and alignment baselines.
- Instruction hierarchy is preserved, with minimal safety tax on general capabilities; unlike blackbox monitoring, performance is consistent regardless of how suspicious the misbehavior looks externally.
- The behavior persists under adversarial prompt optimization and generalizes to settings where agents pursue misaligned goals themselves rather than being instructed to misbehave.
The authors argue this offers a viable path for reducing frontier misalignment risk, assuming neither that misbehavior can be prevented nor that it can be reliably classified from outside. The thread originated from Guy Trask's suggestion of a "Whistleblower AI" paper.
More from Safety
- Untethered iOS 27.0.1 jailbreak demoed on iPhone 18 Pro at Hexacon — jedisct1 · 2026-10-16
- China requires MOFCOM approval to open-source models above 2T params or 10^26 FLOPS — zephyr_z9 · 2026-10-16
- Universal forgery attack breaks Chinese PQC signature Origami using only the public key — matthew_d_green · 2026-10-16
- 400+ researchers sign open letter: AI needs both alignment and containment — ghadfield · 2026-10-16
- Celebrity AI Deepfakes Rack Up Millions of Views as Verified Fan Accounts Spread Fakes — ChuckDBrooks · 2026-10-16
- New Yorker excerpt: Brockman kept asking 'how do we win?' when pitched 2017 AI safety coalition — ben_j_todd · 2026-10-16