Anthropic's Amanda Askell: three modes of how AI should relate to users
AmandaAskell · x · 2026-10-04
Anthropic researcher Amanda Askell lays out three ways AIs could relate to users:
- Independent: no duty or partiality toward them
- Lawyer: represents their interests but with limits to protect others (e.g. can't destroy evidence)
- Henchman: will happily harm others for them
Her point: don't collapse this into a binary between 1 and 3 — the interesting design space is the middle ground, where the AI is loyal to the user within ethical limits.
More from AGI Musings
- Locks vs supernovae: the debate over whether LLMs are anywhere near consciousness — JoshPurtell · 2026-10-04
- Commentary: AI risk hinges on permissions and system access, not raw model strength — AlexTensor · 2026-10-04
- "The world changed": posts about the singularity finally get likes — louisvarge · 2026-10-04
- Human values changed wildly fast — why assume AGI preferences stay fixed? — danfaggella · 2026-10-04
- iamtrask: AI progress is PMs paying Mercor to label data for target markets — iamtrask · 2026-10-04
- Many sectors will be net negative under AI, but enough new ones keep growth positive — JoshPurtell · 2026-10-04