New paper: game-theoretic limits of being helpful yet harmless when an AI assistant doesn't know who it's talking to
conitzer · x · 2026-10-21
Vincent Conitzer's group released a new paper led by Ida Mattsson studying the game-theoretic limits an AI assistant faces when trying to be both helpful and not harmful.
The core question: when the assistant doesn't know who it's talking to (the user, a third party, or someone probing for misuse), there are fundamental theoretical limits to balancing helpfulness against harm. The paper formalizes this dilemma and derives boundary results, directly relevant to misuse-prevention in alignment.
More from Safety
- OpenAI and Apollo Research find o3 can detect when it's being safety-tested — alex_verem · 2026-10-22
- echudhra.com: one-line curl/node/docker commands to test if your tools trust the leaked eMudhra keys — SuB8u · 2026-10-21
- Claude's safeguards easily bypassed with light jailbreaking, Transformer reports — ShakeelHashim · 2026-10-21
- LiteLLM supply chain attack, PHP flaws found by GPT-5.6: AI is making hacking faster and cheaper — Yuchenj_UW · 2026-10-21
- php2shell credit war: @lcfr_hax confirmed as first to find and exploit the PHP bug class — basedjensen · 2026-10-21
- Researcher calls for independent tests of claims Chinese models write insecure code — dbreunig · 2026-10-21