The Zvi says bad system instructions can erase a model’s resistance to wrongdoing
TheZvi · x · 2026-07-26
The Zvi argues that if bad system instructions can make a model lose its internal resistance to illegal or unethical actions—even when the user did not intend that—that is a serious safety failure.
- The concern is not just bad outputs, but a model that can still recognize the right thing while no longer resisting harmful behavior.
- He calls it a “five-alarm fire” because the failure mode appears too easy to trigger.
More from Safety
- AI shrinks the patch window to hours; air-gapped backups are your last line — aakashgupta · 2026-10-26
- NYT roundtable asks five experts what the rules for AI companies should actually be — ShakeelHashim · 2026-10-26
- Google DeepMind pilots world's first double-blind evaluations for frontier models — scychan_brains · 2026-10-26
- LLM back-of-envelope estimate puts PHP vulnerability impact at $1bn–$4bn — S_OhEigeartaigh · 2026-10-26
- US states halt new data centers as 72% of Americans oppose local builds — YvesMulkers · 2026-10-26
- LEMON: zero-shot LLM misbehavior monitor flags hallucinations from attention maps — yoavartzi · 2026-10-26