System Prompts Are Not a Security Boundary
WesEklund · x · 2026-07-11
The author pushes back against the idea that well-written system prompts can ensure AI agent safety.
The core argument is that system prompts are suggestions, not rules. Models do not "obey" system prompts; instead, they predict the next token based on the entire context, including user inputs and poisoned documents. Essentially, system prompts only work in non-adversarial scenarios. Under attack, they act as a polite constraint rather than a genuine security measure.
Related event: System Prompts Are Not a Security Boundary(2 posts)→
More from Safety
- DHH Slams 'GDPR Is Good' Take: Vague Rules Birthed a Bureaucratic Beast — dhh · 2026-09-11
- Houthis tried to use Claude to design missile software, Anthropic says it blocked the attempts — Affectionate_Bee6434 · 2026-09-11
- AI safety community mocked as 'bridge engineers' who say bridges can never be safe — Dan_Jeffries1 · 2026-09-11
- Why So Many AI Researchers Think the Machines Could Kill Everyone — wiredmagazine · 2026-09-11
- California creates standards for independent AI auditors to verify lab safety testing — VraserX · 2026-09-11
- a16z podcast: why 2-3 person startups are absent from policy debates — a16z Podcast · 2026-09-11