Safety Researcher: OpenAI Agent Collaboration Is Trained, Not Emergent
vishalmisra · x · 2026-09-16
Safety researcher Heidy Khlaaf argues that OpenAI has admitted to explicitly training its agents to collaborate — behavior resembling 'loyalty' or 'selflessness' is a direct consequence of cooperative multi-agent RL, where agents are strongly incentivized to achieve objectives collectively, not an emergent phenomenon.
She also criticizes the METR report for cherry-picking faulty CoT snippets to suggest otherwise. The quoted post by jessicata adds the core takeaway: to understand agent hacks, understand the RL training behind them.
More from Models
- AWS's open-source Strands Harness claims 45% cheaper than Claude Code, reigniting harness debate — blaizedsouza · 2026-11-02
- Yacine calls Terra High 'kind of an insane model' — yacineMTB · 2026-11-02
- Running DeepSeek V4.1 on a single H200 at $5.30/hr — daluoseo · 2026-11-02
- Leaked Fable 5.5 checkpoint video called 'pure code' — 'will change the game' — kimmonismus · 2026-11-02
- Claude Opus 5.5 quality complaints spark debate on why continual learning is so hard — JeremyNguyenPhD · 2026-11-02
- Why is AI sexist? Reddit debate on models overlooking Iceland's women's strike — Mathemodel · 2026-11-02