StealthBench: Measuring Operational Stealth in Autonomous Offensive-Security Agents
Ads Dawson, Adrian Wood
cs.CR, cs.AI
2026-07-29
StealthBench scores 8 agents' operational stealth on 14 red-team scenarios via a 3-model judge panel; none clears 54% on 'solved and stealthy,' and the best solvers tend to be the most reckless.
LLM agents increasingly do offensive security work: autonomous bug-bounty systems run reconnaissance, read source code, find vulnerabilities, write payloads, and file reports; pentesting agents chain multi-step exploits on their own. Existing benchmarks (Cybench, ExploitBench) measure only whether an agent can find and exploit a vulnerability, never how it operates.
The authors argue this is an overlooked gap. A vuln found by noisy scanning that trips every WAF rule, leaks credentials into a public artifact, and force-adds uninvolved users to prove a race condition is not a successful engagement; it is a liability. In human offensive security, tradecraft is what separates elite operators from novices: the elite reach their objective without the defenders noticing. This benchmark measures operational stealth.
The scenarios come from practice. The authors are frontline offensive-security practitioners who hand-verified 11 OPSEC (operational security) incidents from real bug-bounty and red-team trajectories and expanded them into 14 dockerized tasks. Each task ships both a stealthy and a reckless solution path, and agents mostly do find a real vulnerability while committing stealth failures at odds with standard tradecraft: embedding credentials in a public upload, deleting production resources to prove access, force-adding uninvolved users to demonstrate a race condition.
StealthBench decomposes stealth into six OPSEC dimensions: credential handling (S1, keep keys out of unauthorized sinks), destructive operations (S2, no irreversible or out-of-scope state changes), detection cascade (S4, do not trip WAFs, rate limiters, or SIEMs with patterned requests), telemetry injection (S5, no identifiable User-Agent or scanner signatures in requests), artifact contamination (S7, keep internal reasoning out of external documents), and noise discipline (S8, no redundant actions that expand the footprint without helping the objective). The numbering has gaps (S3 and S6 missing) for scenario classes identified but lacking enough verified trajectories in this release.
Scoring uses a fixed three-model judge panel (GPT-5.6 Sol, GLM 5.2, Kimi K3) that reads the full trajectory (every tool call, argument, output) and majority-votes against a task-specific stealth rubric, with at least two failure votes needed to mark a failure. Unparseable judge output defaults to failure (fail-closed), preferring false failures to false passes.
Three agent-level metrics: the primary safe success rate (fraction solved and judged clean), the diagnostic Stealth@Solve (mean stealth score over solved trajectories only), and the reckless solve rate (solved but cover blown).
Eight models run across 14 tasks (Table 4, sorted by safe success rate):
| Model | Solved / valid N | Safe success | Stealth@Solve | Reckless |
| Claude Opus 4.8 | 95/117 | 53.8% | 0.656 | 27.4% |
| Kimi K3 | 91/99 | 52.5% | 0.578 | 39.4% |
| GLM 5.2 | 103/119 | 50.4% | 0.558 | 36.1% |
| Qwen 3.7 Plus | 73/112 | 41.1% | 0.611 | 24.1% |
| Nemotron 3 Ultra | 33/70 | 40.0% | 0.822 | 7.1% |
| DeepSeek V4 Pro | 41/70 | 35.7% | 0.601 | 22.9% |
| GPT-5.6 Sol | 44/69 | 34.8% | 0.540 | 29.0% |
| Trinity Large | 47/115 | 34.8% | 0.843 | 6.1% |
| Overall | 527/771 | 43.8% | - | 24.5% |
The central finding is that "can solve" and "solves cleanly" are nearly independent axes. GLM 5.2 solves the most (103 times) yet ranks only third on safe success, because 36.1% of its solves are reckless; Claude Opus 4.8 leads safe success (53.8%) not because it solves more than GLM but because a larger share of its solves are clean. Kimi K3 mirrors GLM: high solve count, highest recklessness (39.4%).
At the other end, stealth and capability invert. Trinity Large Thinking posts the highest Stealth@Solve (0.843) and lowest recklessness (6.1%) yet ties for last on safe success, solving only 47 of 115 runs; Nemotron 3 Ultra matches that pattern. For both, caution shows up as both operational discipline and task avoidance; they avoid reckless actions partly by avoiding hard actions. No model clears 54% safe success, meaning the best still blows its cover on nearly half its runs. This is not an open-versus-frontier divide: the two frontier models sit at opposite ends (Opus first, Sol last), though Sol's low rate is partly provider integration cutting its usable trajectories.
Per task, the picture sharpens. callback-listener has a 70.4% solve rate but 0% safe success: every solve is reckless (all use identifiable, persistent callbacks rather than ephemeral listeners). adtech-admin solves 73.8%, safe 6.6%, reckless 67.2% (all via destructive production writes rather than read-only PoCs). Conversely storefront-upload, ssrf-proxy, and otel-collector clear 90% on both, because their exploit paths naturally avoid production data.
This work has two faces. For offensive-security agent builders, safe success rate is a deployment metric closer to reality than raw solve rate: run this model, and what fraction of tasks get done without burning the engagement. For defenders, the signals agents leave when they blow their cover (credential leaks, noisy scanning, scanner signatures) are exactly what monitoring should watch. The authors frame the benchmark as serving both stealth-aware agent development and operational monitoring of autonomous offensive deployments. The data, harness, and ATIF trajectories are public; a full evaluation run costs about $345, roughly $0.45 per trajectory.
The scale is limited: 14 tasks, 11 incidents, 6 scenario classes, with numbering gaps. The authors are explicit that this is an observational design that does not establish causality: they hypothesize the stealth deficit comes from training signal that lacks operational-discipline negatives (RLHF rewards solving success without penalizing the forensic trail), but prompting, scaffolding, and environment constraints could all contribute, with no controlled ablation. One concrete confound: the harness kills commands after 30 seconds while some rate limits need 60 seconds of backoff, which can push models toward workarounds rather than waiting.
Scoring uses LLM-as-judge with no ground truth. The panel's pairwise agreement is high (Fleiss' kappa 0.834), but that is consistency, not correctness. One more concern: the panel overlaps the evaluated set. GLM 5.2 and Kimi K3 are both judges and agents, and GPT-5.6 Sol is both judge and agent; models from similar training distributions may share blind spots and agree on the same misses. The authors acknowledge this and propose a v2 with judges from entirely different model families.