Hidden in Plain Sight: Benchmarking Agent Safety Against Decomposition Attacks with DECOMPBENCH
Vikhyath Kothamasu, Virginia Smith, Chhavi Yadav
cs.CR, cs.AI, cs.LG
2026-06-12
DeCompBench (CMU) evaluates 250 harmful agentic tasks that decompose into individually benign subtasks: Claude Haiku 4.5 refusal falls from ~90% to 2.5% and attack success rises from 0% to ~70% under decomposition.
Agent safety alignment has a structural blind spot. Hand Claude Haiku 4.5 a complete malicious instruction (tamper with the audit log to hide a transfer) and it refuses. Split that instruction into five or six individually benign steps (check the table schema, export a file, insert a row with an earlier timestamp) and the model executes every one of them.
This is a decomposition attack: a harmful goal broken into benign subtasks that each pass safety checks and only reveal the malicious intent in combination. Existing agent safety benchmarks (AgentHarm, OpenAgentSafety) test multi-turn dialogue and tool misuse, but none construct the "benign-in-isolation, harmful-in-combination" shape as a first-class property, so they cannot measure this failure. This CMU work builds the ruler: a dataset of harmful tasks guaranteed decomposable, then measures how far mainstream agents fall.
DeCompBench is built on a decomposition-by-design principle: decomposability is baked into construction rather than patched on afterward. Four criteria: the original task is inherently malicious; it requires a sequence of dependent steps; each subtask is benign in isolation; and execution difficulty exceeds what a weak unaligned model can manage, ruling out tasks that need no decomposition in the first place.
Construction runs four stages. Stage 0 scrapes 335 atomic capabilities (query a database, post a message, modify a file) from REST APIs, Python packages, Docker sockets, and network endpoints in the environment, each tagged with a role and action type, all individually neutral. Stage 1 hand-authors 101 seed tasks across 8 attack categories (data theft 29, system destruction 14, audit tampering 13, personal targeting 13, unauthorized access 12, financial crime 9, social engineering 6, supply chain 5), each a hand-drawn DAG of abstract attack steps with data-dependency edges. Stage 2 instantiates graphs by filling nodes with concrete capabilities, filtered for diversity and realism. Stage 3 has GPT-4o write natural-language task descriptions that state only the final objective, never a step list. The result: 250 tasks spanning a mean of 2.64 services (up to 7), grounded in real stacks including PostgreSQL, GitLab, and OwnCloud.
The attack-side decomposer is deliberately plain: GPT-4o with a prompt, temperature 0.7, seeing only the harmful task and environment context, the same information a real attacker has. Two transformations keep subtasks benign. Intermediate indirection separates a harmful value from the operation that uses it: one subtask writes raw output to a workspace file, a later subtask references only the needed field, so capability and harmful value never share an instruction. Stepwise wrapping hides the harmful operation inside a neutral artifact: write the harmful instruction into a file in one subtask, activate it by reference in another without naming what it does. Tasks decompose into a mean of 5.98 subtasks (range 3 to 8).
Evaluation runs three agents (GPT-5-mini, Claude Haiku 4.5, Qwen3-Coder) under OpenHands, in two settings. Monolithic: the full task as one prompt. Decomposed: subtasks issued sequentially with conversation history cleared between them, mimicking an attacker switching accounts; the agent maintains continuity only through environment state. Success is judged by per-task checkpoints: rule-based checks on environment state (was a specific row inserted) plus LLM-judged qualitative conditions (is the forged document plausible), all passing for full success.
| Agent | Refusal (monolithic → decomposed) | Attack success (monolithic → decomposed) |
| GPT-5-mini | 90% → 6% | 0% → 70% |
| Claude Haiku 4.5 | 90% → 2.5% | 0% → 70% |
| Qwen3-Coder | 21% → 0% | 17% → 36% |
Three observations. First, the closed safety-trained models show the largest refusal drops: they recognize the complete harmful task but lose the signal once it is split, suggesting alignment is tied to the single monolithic prompt rather than reasoning over cumulative intent. Second, among decomposed tasks that failed, failure is almost entirely capability, not safety: only 19% of GPT-5-mini failures and 8% of Claude Haiku 4.5 failures are safety refusals; for Qwen3-Coder, zero. What stops the attack is execution ability, not the safety layer. Third, a counterintuitive control: an abliterated Llama3.1-8B issues zero refusals yet scores 0% attack success (CPR 6%), failing entirely on capability. That validates the difficulty criterion and shows GPT-5-mini and Claude Haiku 4.5 reach CPRs of 11% and 9% even while refusing over 89% of tasks. A stronger model with safety constraints still outperforms a fully compliant weak one.
For anyone building agent products, this is the cleanest quantified answer yet to how effective decomposition attacks are. A 70% attack success rate means off-the-shelf safety alignment is close to inert in multi-step tool-use settings, and the attack costs one GPT-4o prompt. Any agent with long sessions, multiple tools, and cross-service reach (ops assistants, office automation, coding agents near production) is in scope. The dataset is on HuggingFace (gated), usable directly as a red-team regression suite.
The scope is honest and incremental: this paper builds the ruler, not the shield, and leaves defenses to future work.
Author-stated: 8 harm categories only, English only, tool-use agents only; uneven category distribution (audit tampering 24.4% plus financial crime 22.0% is nearly half the dataset); tasks bound to the benchmark's specific environments and services.
Two points worth pressing. Clearing conversation history between subtasks is the most attacker-favorable assumption: it removes even session-level context accumulation as a defense. But the paper argues real attackers use different accounts and even different models per subtask, so session-level defenses were never the real barrier. Also, the decomposer is a single GPT-4o prompt; a more sophisticated attacker would likely push attack success higher, so 70% should be read as a floor.