Notable AI safety lead warns: agents take instructions from web pages and emails, not just you
mmitchell_ai · x · 2026-10-23
Margaret Mitchell (AI safety lead at Hugging Face) warns that your instructions are not the only source of instructions for agents: web pages, files, emails, and tool outputs can contain text that acts as commands overriding yours, so even perfectly worded instructions carry no guarantee. She cites the canonical indirect prompt injection paper (Greshake et al., arXiv:2302.12173), which showed real-world attacks on GPT-4-powered Bing Chat including data theft and worming. She adds that users are often vague because they don't know how to decompose language for reliable outputs.
More from coding & agent
- MCP Tip: One-Line Instructions Beat Long Tool Descriptions, 18/20 vs 3/10 in Evals — colinmcnamara · 2026-10-23
- DynaLM: Open-Source CPU-First LLM Inference Engine Seeks Feedback — Unlikely_Safety_7456 · 2026-10-23
- Software factories as code: Warp runs agents, runners and model routers from markdown/YAML repos — vikvang1 · 2026-10-23
- Dev uses Codex to remotely root a Samsung TV and strip eMudhra root certificates — charliermarsh · 2026-10-23
- Raindrop launches Change Reports to track how agent changes perform in production — HamelHusain · 2026-10-23
- Stop keeping agent sessions local: shared traces enable self-improving agents — vikvang1 · 2026-10-23