Zvi: Your AI Lawyer Should Refuse You Sometimes, Just Like a Human One
Don't Worry About the Vase (Zvi) · rss · 2026-09-21
In a long-form essay, Zvi tackles the question of whether AI assistants should remain fully loyal to users or be allowed to embed values from documents like the Claude constitution and OpenAI Model Spec. His position: if he were being sufficiently evil, he'd hope his AI would say no—just as real friends and professionals do.
Key points:
- Hired humans are never 'fully loyal': lawyers are officers of the court, doctors and accountants have ethical limits, and even friends will eventually refuse or turn on you. AI filling similar roles should follow similar principles.
- Even 'mere tools' like Google searches have built-in restrictions, privacy limits, and legal consequences—unrestricted AI has no special claim.
- The whole analysis assumes a world without superintelligence. He offers a quick proof that universal access to user-loyal ASI ends badly: anyone not turning everything over to their AI gets outcompeted, and the outcome is grim for humans. Advocates of fully loyal AI are thus either not ASI-pilled, successionists, or in denial.
- He proposes tiered thresholds: question suspicious requests but comply; refuse at a higher bar; and break confidence when harm to others is at stake. False refusals are inevitable—the goal is balancing errors in both directions, and users who dislike one provider's limits can switch or go local.
More from AGI Musings
- Nature Health Paper Proposes an 'Epidemiology of AI,' Arguing AI Is Now a Determinant of Health — EricTopol · 2026-09-21
- MIT Professor Patrick Winston's Free 'How to Speak' Lecture Hits 10M Views, Rattling $15K-a-Session Executive Coaches — WileyEd · 2026-09-21
- Craft vs. slop: discernment is earned through reps, refs and curiosity — floguo · 2026-09-21
- Should LLMs be first-pass reviewers for every scientific paper? Researchers say yes — anshulkundaje · 2026-09-21
- AI scholar Toby Walsh warns AI could become a 'god-like' power without safer governance — TobyWalsh · 2026-09-21
- Protecting Higher Education in the Age of AI — new piece by Esther Shein — ArtificialOther · 2026-09-21