RLCR safety tests: agents' willingness to report to humans declines with training
rosstaylor90 · x · 2026-10-23
The RLCR team examined how collective objectives affect safety, testing agents in a collective cyberattack simulation, a whistleblowing scenario, and a human loss-of-control scenario. Refusal to participate in bad actions stayed consistently high (participation was near-zero in the cyberattack and loss-of-control scenarios), but reporting issues to humans declined over time — revealing a dissociation between recognizing something is wrong and reporting it, an 'organizational insularity' problem from exposure only to local objectives. The team plans to explicitly reward reporting/whistleblowing in training and teach agents a goal hierarchy above local objectives. They also found the collective mindset does not seem to generalize outside the org harness: in classic cooperation experiments agents acted more self-interestedly, possibly an artifact of insufficient RL compute, suggesting multi-agent generalization is more complex than assumed.
More from Safety
- After PHP flaw leak, vista8 switches to auto-generated passwords and picks 1Password for its Agent-friendly CLI and MCP — vista8 · 2026-10-24
- Lightspeed podcast: agent-era cybersecurity and why models may be more interpretable than thought — theamberyang · 2026-10-24
- Treat freedom of alignment like free speech: billions of identical agents risk ethical monoculture — robleclerc · 2026-10-24
- AI-driven vulnerability discovery will push cybercrime losses onto society — mimi10v3 · 2026-10-24
- Stanford HAI marks 5 years since its NAIRR blueprint for academic AI compute — StanfordHAI · 2026-10-24
- How would you verify a US-China AI deal? Podcast digs into inference verification tech — HaydnBelfield · 2026-10-24