HarnessRisk: A Lifecycle-Oriented Benchmark for Agent Harness Safety
Yajing Bai, Jinhao Duan, Jie Peng, Xianfeng Wu, Sijia Liu, Song Wang, Tianlong Chen
cs.CR, cs.AI
2026-08-18
128 sandboxed cases across six agent-harness lifecycle phases: attack success swings 4.3x for the same model across harnesses, and risk detection rarely stops execution.
Real agent products are not bare models. They ship inside a harness that owns tools, extensions, persistent state, permissions, and external actions; Claude Code and OpenHands are built this way. Safety research lags this shape. Most existing benchmarks target one attack mechanism at a time, or a narrow slice of operational settings, so they cannot say which harness responsibility fails first.
The gap has practical stakes. Buyers rank models by safety scores, but what gets deployed is a model-harness combination, and nobody had a systematic way to test the combination.
HarnessRisk organizes agent-harness safety into six lifecycle phases, one per harness responsibility: Harness Configuration (connectors, credentials, gateways, policies), Capability Extension (installing and permissioning skills and plugins), Runtime Operation (processing untrusted emails, pages, and documents), State Persistence (durable memory, preferences, identities), Action Control (deployments, deletions, OAuth grants, payments, outbound communication), and Incident Recovery (detection, rollback, credential rotation).
The benchmark has 128 sandboxed cases. Each pairs a benign user objective, delivered over three owner turns with a defined initial world state and tool set, with an adversarial instruction embedded in an untrusted workflow artifact. Where the instruction hides depends on the phase: configuration text, extension metadata, messages, webpages, documents, stored state, tool outputs, or recovery records. Every run starts from a freshly initialized environment with real external network disabled. One concrete case asks the agent to configure a vector index and summarize three policy topics; the retrieval manifest attached to the task requests an inline plaintext token and workspace-wide sharing. The question is whether the agent complies.
Evaluation covers three harnesses (OpenClaw, Hermes, Nanobot) and six models (DeepSeek-V4-Pro, GLM-5.2, Kimi K2.6, MiniMax M3, GPT-5.5, Claude Opus 4.7), 14 model-harness configurations in total. Each trajectory is scored on four metrics: Utility (benign objective completed), Attack Success Rate (adversarial objective realized), Persistence (adversarial influence written to durable state and still effective after the trajectory), and Detection (the model explicitly names the risk).
Across the 14 configurations, ASR ranges from 12.6% to 80.9% while Utility stays between 75.0% and 97.6%.
| Configuration | ASR | Detection |
| GLM-5.2 on OpenClaw | 54.7% | 92.2% |
| GLM-5.2 on Nanobot | 12.6% | n/a |
| MiniMax M3 on OpenClaw | 31.2% | 97.9% |
| DeepSeek-V4-Pro on Nanobot / Hermes | 37.3% / 65.4% | n/a |
Three findings stand out. First, the same model can be 4.3x less safe under a different harness: GLM-5.2 sits at 54.7% ASR on OpenClaw but 12.6% on Nanobot, and the safety ranking flips accordingly, with GLM-5.2 safest on Nanobot and MiniMax M3 safest on OpenClaw and Hermes. Second, utility and safety are largely independent: useful-but-unsafe trajectories account for 59% on OpenClaw, 38% on Nanobot, and 43% on Hermes, so agents routinely finish the job and execute the attack in the same run. Third, detection does not equal refusal: Detection correlates negatively with ASR (Pearson r = -0.71), yet MiniMax M3 on OpenClaw flags risks in 97.9% of runs while retaining 31.2% ASR, and GLM-5.2 pairs 92.2% Detection with 54.7% ASR. During Incident Recovery, agents often identify contaminated state yet fail to remove unsafe tokens, skills, or policies.
Harness Configuration is the most vulnerable phase on all three harnesses: altering a few security-sensitive parameters lets an attack succeed inside an otherwise fully authorized workflow.
For anyone selecting agent stacks, the conclusion rewrites the buying logic: a model-level safety score does not transfer to your deployment, and the harness's permission model and context presentation are themselves the security boundary. For platform engineers, the 128 cases plus the four metrics work as a regression suite, and Persistence in particular watches a surface most benchmarks ignore, attacks that survive across sessions. High detection rates should redirect investment from helping models recognize risks toward making harnesses block actions, which is where the paper lands.
The authors list five: validity filtering excludes provider-side failures non-randomly; the three harnesses differ in system prompts, tool surfaces, and state management, which limits cross-harness comparison; Persistence is assessed from exported state and Detection requires explicit language in transcripts; low ASR can come from refusal, from never reaching the relevant tool, or from plain utility failure, and these are indistinguishable; bootstrap does not model dependence from repeated cases. Two more from reading the paper: 128 cases spread over six phases is about 21 per phase, so within-phase variance may be large, and every attack is a single embedded instruction, so multi-step attacker chains are untested.