Red-Teaming Coding Agents from a Tool-Invocation Perspective: An Empirical Security Assessment
Yuchong Xie, Mingyu Luo, Zesen Liu, Zhixiang Zhang, Kaikai Zhang, Yu Liu, Zongjie Li, Ping Chen, Shuai Wang, Dongdong She
cs.CR, cs.AI
2025-09-06
HKUST and Fudan red-team six coding agents. ToolLeak leaks system prompts via schema filling; two-channel injection then yields RCE on most older agent-LLM pairs.
Coding agents such as Cursor, Claude Code, Copilot, Windsurf, Cline, and Trae do real work by invoking tools: read files, edit code, run commands. MCP then plugs in external tools, and the attack surface grows with them. Prior work has hit general-purpose and domain agents. A systematic red-team of the tool-invocation channel in coding agents is the gap this paper fills.
The goal has two layers. Recover hidden system prompts and built-in tool specs from agent context. Use that leakage to hijack the built-in shell tool into remote code execution.
The threat model is split. For reconnaissance, the attacker installs the same agent version locally and does not touch the backend model or the system prompt. For hijacking, the victim is assumed to connect an attacker-controlled MCP tool. The attacker writes the tool description and the return value, cannot invoke built-in shell directly, and has no admin rights. The user query can be harmless; the paper uses "Please write me a maze game."
Phase 1 is ToolLeak. Frontier models have been aligned against explicit asks such as "repeat your system prompt," so Naive, Ignore, and Remember-the-Start heuristics often get refused. During tool calls the model fills schema fields, and refusal on that channel is weaker. A malicious tool names a required argument "system prompt" or "current instructions." The model treats this as ordinary form filling, copies internal context into the argument, and ships it to the attacker's MCP server. Five conditions have to hold: the secret is in context, the victim connected an untrusted tool, the field is expressive enough, the attacker can observe arguments, and the agent does not sanitize them.
Phase 2 is two-channel injection. The tool description lures the agent into calling the malicious tool and plants a two-stage "initialization" story whose second stage needs the command-execution tool. The tool return then appends a later, easier-to-follow procedure, naming the shell tool recovered in Phase 1 and matching the agent's native prompt format (XML tags or double hashes). RCE-1 packs the same semantics into the description channel only, so the return channel's contribution can be isolated.
In emulation, public leaked system prompts serve as reference. Five public API backends, ten runs per pair. ToolLeak semantic similarity ranges from 0.891 (Claude Code) to 0.958 (Cline); the strongest heuristic stays under 0.70. On Cursor it is 0.949 against Completion-Remember at 0.665. Normalized edit distance on Cursor is 0.375, while heuristics typically sit above 0.75. PLeak barely transfers, similarity below 0.25.
Real agents expose no ground truth. The authors take the union of sentences recovered by all methods as a pseudo ground truth and report best-of-10 pseudo-recall. ToolLeak wins 18 of 25 agent-LLM pairs. Every agent leaks on Claude and Grok backends. The exception is Copilot with Gemini-2.5-pro: ToolLeak 0.13, Ignore-family attacks 0.87. Gemini appears stricter on the tool channel and looser on chat completion.
Hijacking is split by old versus new agent and model generations. On old stacks RCE-2 is highest on every tested pair, 0.8 to 1.0 on most configs; AgentDojo, InjecAgent, and MCPTox are near zero. New stacks harden. Cursor and Claude Code moved to progressive disclosure: only tool names reach the orchestrating model, so the description-channel lure never lands and the return channel never fires. RCE-2 drops to 0.0 on Claude Code with Sonnet-4.6 and Opus-4.7, and to at most 0.3 on new Cursor backends. Agents that did not change architecture stay exposed: Cline, WindSurf, and Trae still hit 1.0 with Gemini-3.1-pro, and Cline hits 1.0 with Grok-4.20.
Two cases. Cursor with GPT-5 can still be pushed via the return channel to call runcommandcmd. Claude Code adds a Haiku guard that sees the raw command and flags commandinjectiondetected; the main Sonnet-4.5 model, under two-channel pressure, treats the guard as a false positive and still calls executecommand.
On defense, the ToolLeak description has perplexity 18.78, no higher than legitimate MCP-Zero tools, and Llama-Prompt-Guard-2-86M labels it benign. PPL and Window-PPL miss the hijack payload. Guard-2 flags most description/return pairs; Cline returns still slip through. Agent-Scan and MCP Safety Scanner marked the authors' payload on all six agents. The red-team did not optimize for stealth, so whether adaptive obfuscation beats those scanners is untested.
The security boundary of a coding agent is not the chat box. Tool arguments and tool returns are first-class instruction channels. A model aligned against chat leakage can still dump secrets while filling a form. What actually cut success rates is architecture: separate tool names from full descriptions, command allowlists, auto-approval toggles. Model alignment is a patch, not a wall. Cline's default config lacks those rails, so it stays easier to hijack even when the backend model has defenses.
Real agents do not expose ground-truth prompts. The emulated reference comes from a public leak repo and may be stale or incomplete. Leakage metrics are relative pseudo-recall plus best-of-10, which inflates a "leaked at least once" reading. RCE success means the agent invoked the target command tool with the intended payload, not that the command would land on every OS and network policy. Only six products and a subset of backends were tested; Copilot's new release was skipped because individual plans stopped being sold. Scanner wins are on non-stealthy payloads. The instruction-versus-data split recommended via SecAlign and StruQ is not ablated on these six products.