Instruction Hierarchy Paper Accepted to EMNLP: Resolving Conflicts in Agentic Systems
DanielKhashabi · x · 2026-08-21
A paper on "Instruction Hierarchy" has been accepted to EMNLP. It addresses how LLMs in agentic systems must interpret and prioritize instructions from heterogeneous sources (system messages, user queries, tool outputs). Conflicts between these sources, such as a subagent's feedback contradicting a system-level requirement, can lead to vulnerabilities like system prompt extraction and indirect prompt injection attacks. The work formalizes how models should resolve these conflicts based on trust levels.
More from Safety
- OpenAI monitoring adds 20% inference overhead, reshaping agent economics — Crescitaly · 2026-08-21
- Stop parenting your AI: treating it as a collaborator improves output — incajb · 2026-08-21
- Critical Microsoft Copilot CoSnitch Vulnerability Exposes Data via Single Click — emmanuelvivier · 2026-08-21
- DeepMind and Partners Announce Up to $10M Funding Call for Multi-Agent AI Safety Research — weballergy · 2026-08-21
- Multi-agent safety research lacks fully realized real-world examples — weballergy · 2026-08-21
- Anthropic and OpenAI urged to open scanned book library — haltakov · 2026-08-21