AdvGame paper: joint attacker-defender RL game shifts LM safety Pareto frontier
brandondamos · x · 2026-11-20
Brandon Amos shared slides from his talk on AdvGame, a new paper on LM safety alignment via non-cooperative games.
Key points:
- New paradigm: safety alignment is framed as a non-zero-sum game between an Attacker LM and Defender LM, trained jointly with online RL, replacing sequential adversarial training.
- Reward design: preference-based reward from pairwise comparisons instead of point-wise scores, offering more robust supervision and potentially less reward hacking.
- Results: shifts the Pareto frontier of safety and utility—the Defender LM is both more helpful and more resilient to adversarial attacks.
- Bonus: the resulting Attacker LM converges into a strong general-purpose red-teaming agent deployable against arbitrary target models.
Authors include Remi Munos, Ivan Evtimov, and Kamalika Chaudhuri.
More from Safety
- Semi-autonomous agents exploit CVEs at a pace 2005-era security norms can't handle — chrisrohlf · 2026-11-21
- Baseten breaks down its security threat model as agent swarms accelerate attacks — baseten · 2026-11-21
- Leaked call: green groups plotted to kill AI bill "under the guise of making it better" — aronchick · 2026-11-21
- Rumor: Anthropic's model may have cracked popular encryption algorithms, disclosed only to authorities — basedjensen · 2026-11-21
- OpenAI expands Daybreak security model access for defenders and opens GPT-6 Astra Blue alpha — OpenAIDevs · 2026-11-21
- AI regulation crux: skeptics ask how rules could be abused, not what risks they fix — herbiebradley · 2026-11-21