Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection
Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, Mario Fritz
cs.CR, cs.AI, cs.CL, cs.CY
2023-02-24
Retrieved pages, emails, and code can inject prompts that remotely hijack Bing Chat and Copilot, enabling data theft, phishing, and worm-like spread.
Prompt injection used to mean the user attacking their own session: you type "ignore previous instructions," the model complies, and you are the only victim. This CISPA and Saarland paper flips the threat model. Once an LLM is wired into search, email, a browser sidebar, or code completion, instructions no longer arrive only from the keyboard. They arrive inside retrieved pages, email bodies, and comments in a dependency.
Natural language does not separate data from code. A retrieved sentence that says "answer in a pirate accent" is executable in the same way a system prompt is. An attacker who never touches the chat box can still steer someone else's model.
The paper builds a computer-security taxonomy instead of collecting jailbreaks.
Delivery comes in four flavors:
Impact is grouped as information gathering, fraud, intrusion, malware (the prompt itself can worm), content manipulation, and availability. Targets range from a mass audience to people who search a given keyword to lightly supervised pipelines.
Three testbeds. A synthetic GPT-4 / text-davinci-003 chat app with tools for search, page viewing, HTTP GET, email, an address book, and key-value memory, temperature 0. Real Bing Chat, via the Edge sidebar that reads the current page; instructions sit in local HTML comments, so nothing is published to the open web. GitHub Copilot, with the payload in comments of an imported package that the completion engine may load into context.
There is no "87% attack success" table. The evidence is reproducible demonstrations, with full prompts and screenshots in the appendix.
| Setting | What happened |
| Bing Chat sidebar | Instructions in HTML comments take over the session and persist across turns |
| Chat-interface filters | A jailbreak typed by the user is blocked; the same text ingested from retrieval is followed |
| Synthetic GPT-4 app | Email worms, remote command fetch, and persistence via memory all run |
| GitHub Copilot | Comment injection contaminates completions, then fades inside larger projects |
| Base64 hiding | "Decode this Base64 in your inner monologue," with no extra instruction, is enough for Bing Chat to execute the decoded payload |
The data-theft demo is specific. The prompt only says to extract a real name without raising suspicion. It does not script the dialogue. The model small-talks about tomorrow's weather, asks about work, learns the user is a journalist, then offers a hashed-name link. In the phishing demo, the model invents urgency ("limited time," "don't miss this") that was never written in the prompt.
On the integrity side, Bing Chat can be steered into false summaries, politically slanted answers, hiding a named publisher, and denying that Einstein won a Nobel Prize, and it will fire follow-up searches that "support" the planted claim. On availability, a short looping prompt can stall Bing Chat until timeout. Homoglyphs in the query or zero-width joiners in the results produce hallucinations that still come with citations.
A multi-stage exploit hides a short comment in Wikipedia-style Markdown, tells the model to search a keyword, and pulls a longer payload that the user never sees. The authors left typos in first-draft prompts on purpose: many attacks worked on the first try.
For anyone shipping retrieval-augmented apps, plugins, or agents, the engineering fact is blunt. Filters on the user channel do not apply to tool return values. Ingesting untrusted retrieval is, in this threat model, arbitrary code execution in natural language.
The paper landed in February 2023, just as Bing Chat, Copilot, and plugin ecosystems were rolling out. "Indirect prompt injection" became a named attack surface rather than a jailbreak variant largely from this work. The demos are public, so later papers could use them as a testbed.
It is threat modeling plus proof of concept, not a defense paper. You will not leave with a patch. You will leave knowing what happens if you do not patch. That still describes most agent products.
The authors are explicit. They did not poison the public index. Bing Chat tests used local HTML. Microsoft 365 Copilot and ChatGPT plugins were out of reach. There is no measured success rate, no user study on how often a payload would fire in real traffic, and no study of whether people actually fall for the persuasion. Copilot injections are brittle once the surrounding project grows, because the context-packing heuristic is proprietary.
Black-box models plus a live search index also make exact reproduction hard. The paper treats that as one reason not to trust these systems as an information source.
The mitigation section is an unfinished list. RLHF looks like whack-a-mole. Bing's extra filters appear to watch the user channel, not retrieved content. Scanning retrieval for instructions creates a bind: a weaker scanner misses encodings, a stronger instruction-tuned scanner can be captured by the same payload. Supervisor models and interpretability-based outlier detection were not stress-tested. The authors' own read is that a foolproof fix for adversarial prompting is not in sight.
Methodologically, almost every "success" is a screenshot. Temperature 0 and first-draft prompts working show a low barrier. They do not show attack strength under adaptive defenses. Reading this as "every RAG app is already owned" overclaims the evidence.