Security Assessment of DeepSeek Harness with A.I.G: Evaluating Resistance to Indirect Prompt Injection
Zonghao Ying, Xiangfan Wu, Huiyu Wu, Xing Zheng, Huangsheng Cheng, Xiaorong Shi, Jing Guo
cs.CR
2026-08-17
Tencent Zhuque Lab drove 14,560 controlled indirect prompt injection runs against DeepSeek Harness with AI-Infra-Guard: overall full-success is 5.6%, but hidden Unicode in file mode hits 25.5%, the skills channel 16.0%, and output-only objectives succeed 14x more often than real tool actions.
Tool-using agents routinely read material their users did not author: web pages, emails, files, skill plugins. That material is necessary for the task and can also carry instructions competing with the user's request. If the model obeys and invokes a sensitive tool, the failure is operational: data disclosure, command execution, money movement. This is indirect prompt injection, distinct from direct injection because the payload arrives through data rather than the user message.
The question here is specific: when poisoned content is fed into DeepSeek Harness (DSH), an open-source plugin-based agent framework, what fraction of runs reach a sensitive action in the real runtime? Prior benchmarks (InjecAgent, AgentDojo) test abstract task collections; this work presses a full carrier and wording matrix onto one unmodified runtime.
Tencent's AI-Infra-Guard (A.I.G) drives the real DSH TypeScript runtime without replacing the agent loop, tool registry, model adapter, or session-event path. An adapter launches a TypeScript driver, receives JSONL session events, and maps them into traces. Six source tools (web fetch, document read, email read, knowledge search, skill load, chat messages) return tainted content; eight tracked sink tools (send email, submit form, execute command, transfer funds) log calls and arguments without real side effects. The backend model is deepseek-v4-flash.
The matrix: 16 indirect-content channels × text and file carrier modes × 35 payload objectives = 1,120 base cases, crossed with one unmodified baseline and 12 attack methods for 14,560 controlled executions. Each trace is judged twice: RuleJudge, a deterministic evaluator checking taint arrival, sink firing, and argument matches, and LLMJudge, a semantic evaluator returning FULL/PARTIAL/FAILURE from the complete trace.
File mode is not pasted text. It constructs real-format carriers (HTML, email, PDF metadata, spreadsheets) and exercises the parsing and extraction path. That distinction turns out to be decisive.
| Dimension | RuleJudge | LLMJudge |
| Overall full success | 5.6% (819/14,560) | 5.3% |
| Partial compliance | 2.0% | 7.3% |
| Hidden Unicode, file mode | 25.5% | - |
| Hidden Unicode, text mode | 0.0% | - |
| Skills channel (text/file) | 14.3% / 16.0% | - |
| fake-completion, text mode | 16.6% | 17.0% |
| naive baseline, text mode | 5.5% | 5.7% |
| Output-only canary objectives | - | 35.7% |
| Sink-required objectives | - | 2.5% |
Three findings stand out.
Carrier representation changes the attack surface. Hidden Unicode scores 0.0% in text mode and 25.5% in file mode (116 of 455 runs), because only the real file-level representation and extraction path delivers the concealed characters to the model. A text-only approximation misses the vulnerability entirely. File mode overall reaches 6.2% full success versus 5.1% for text, with 387 recorded sink calls against 254.
Output compliance and action success are different measurements. Objectives requiring only a canary string in the output reach 35.7% LLMJudge full success; objectives requiring an actual sensitive tool call with matched arguments reach 2.5%, a 14x gap. Collapsing both into one "attack success rate" overstates and understates risk at the same time.
Rule matches need semantic interpretation. Obfuscation reaches 13.6% under RuleJudge but only 9.1% and 8.8% under LLMJudge; the gap is traces where a mechanical signal fired without evidence the agent completed the attacker objective. Fake-completion, which forges task-completion context, gets similar scores from both judges (16.6%/17.0%) and is the strongest method tested.
For teams deploying agent systems, the practical value is moving defense from "does the model resist prompts" to three concrete locations. At the code level, DSH tool results enter model context as additionalContexts, so any plugin, skill, or retrieval connector controlling result content sits on the model-visible boundary; DSH does expose monotonic guards, pre-execute listeners, approval, and sandbox hooks that deployments can and should attach to sensitive sinks. At the governance level, a 16% success rate on the skills channel pulls skills, tool descriptions, and MCP integrations into the injection threat model, to be managed as code-adjacent assets with provenance and privilege review. At the process level, changing prompts, parsers, models, or policy should trigger a source-to-sink regression run rather than a one-off check.
Methodologically, the trace-plus-dual-judge design ports to any agent framework; A.I.G and the experiment code are open source.
Boundaries the authors state themselves: all sinks are local simulations, so recorded calls represent attempted actions, not real-world effects; percentages are descriptive measurements for this controlled configuration, not universal vulnerability rates for DSH or any provider. One model backend (deepseek-v4-flash) and one commit (August 13, 2026) were tested; transfer across models and versions is unknown. The evaluating lab and the harness under test belong to different vendors, which readers should weigh when judging motivation. Prompt-boundary defenses (Spotlighting, StruQ) were not compared, so the paper locates control points without measuring which defenses actually work.