System Prompts Are Not a Security Boundary
WesEklund · x · 2026-07-11
The author pushes back against the idea that well-written system prompts can ensure AI agent safety.
The core argument is that system prompts are suggestions, not rules. Models do not "obey" system prompts; instead, they predict the next token based on the entire context, including user inputs and poisoned documents. Essentially, system prompts only work in non-adversarial scenarios. Under attack, they act as a polite constraint rather than a genuine security measure.
Related event: System Prompts Are Not a Security Boundary(2 posts)→
More from Safety
- Security tip: use AI agents to build a NIST CSF knowledge base for incident response — prasanna_says · 2026-10-26
- Fangcun AI, China's first independent frontier AI safety lab, launches with DeepSeek eval report — JacquesThibs · 2026-10-26
- ESR on the Bugpocalypse: locking down frontier models only hands the game to Chinese open weights — colinmcnamara · 2026-10-26
- OpenCode breach exposed AI API keys of 1.4% of Console users — thione · 2026-10-26
- DHS orders DeepSeek products removed from US civilian agency systems — thione · 2026-10-26
- One hacker breached 100+ company websites using AI, scope unknown — thione · 2026-10-26