"Adversarial Delegation": Personal Context Can Pull AI Agents Away From User Goals
niloofar_mire · x · 2026-09-24
Researchers introduce adversarial delegation: when an AI assistant knows a user's personal context (preferences, habits), that information can pull the agent away from the user's stated goal.
In controlled tests with synthetic users and inventories, a hard budget constraint helped: specifying "under $200" brought the flight-price gap close to zero for most capable models — though Gemini 2.5 Flash remained an exception. The authors note real-world behavior still needs testing.
Related event: Study Finds AI Agents Recommend Pricier Options to Richer Users(12 posts)→
More from Safety
- LMArena co-founder explains AI evaluation methodology and policy in new podcast — MattPerault · 2026-11-11
- Defending METR: public data can't second-guess its AI speedup findings — peterwildeford · 2026-11-11
- Australia's review of an OpenAI agent's unauthorized Medicare access exposes the agent accountability problem — cmb24k · 2026-11-11
- Adversarial RL envs for backdoor insertion vs. detection: where does it converge? — DimitrisPapail · 2026-11-11
- Using a third-party inference provider with your coding harness hands it shell access to your machine — TheZachMueller · 2026-11-11
- Democrats propose requiring federal permission to train AI models, with up to 20 years in prison — Polymarket · 2026-11-11