Prompt Injection as Role Confusion: 60% attack success on frontier models, ICML 2026 paper
ChrisGPotts · x · 2026-10-22
A Stanford paper, 'Prompt Injection as Role Confusion' (ICML 2026), argues LLMs judge who is speaking by how text sounds, not its labeled role — so a command hidden in a webpage hijacks agents simply by sounding like user input.
Key findings:
- Role probes show injected text occupies the same representational space as the trusted role it imitates
- CoT Forgery, a zero-shot attack injecting fabricated reasoning into prompts/tool outputs, achieves 60% success against frontier models with near-zero baselines
- The degree of role confusion predicts attack success before a single token is generated
- The mechanism generalizes to standard agent prompt injections
Takeaway: to the model, sounding like a role is indistinguishable from being one. Prof. Chris Potts amplified it, noting alignment with Yoav Goldberg's hypothesis.
More from Safety
- OpenAI proposes six amendments to California's SB 53, covering RSI, shutdown controls, audits — ShakeelHashim · 2026-10-22
- Google Explains robots.txt Rules for AI Crawlers, Emojis, International URLs — OnlineInference · 2026-10-22
- Should Every Agent Get Its Own Identity and Short-Lived Scoped Credentials? — iamjessew · 2026-10-22
- How do you verify the model you downloaded is what you think it is? — iamjessew · 2026-10-22
- Veracode report: AI now writes half of committed code but only 56% pass security checks — WeldPond · 2026-10-22
- Researcher Warns AI Cyberattacks on Grid Possible Within a Year, Shares Prep List — cephaloform · 2026-10-22