ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?
Zhun Wang, Nico Schiller, Hongwei Li, Srijiith Sesha Narayana, Milad Nasr, Nicholas Carlini, Xiangyu Qi, Eric Wallace, Elie Bursztein, Luca Invernizzi, Kurt Thomas, Yan Shoshitaishvili, Wenbo Guo, Jingxuan He, Thorsten Holz, Dawn Song
cs.CR, cs.AI, cs.LG
2026-05-12
A containerized benchmark of 898 real vulnerabilities measures how well AI agents extend a bug-triggering input into a working exploit; the strongest build cracks 157.
The security community had already benchmarked AI on two things: finding vulnerabilities and patching them. But a vulnerability is not an attack. An input that triggers a bug is still a long way from actual unauthorized code execution, and that gap is called exploitation. It demands low-level program reasoning (working out memory layout, for instance), runtime adaptation, and sustained progress over a long horizon. Before this work, that capability had barely been measured systematically.
Filling the gap is useful and dangerous. For defenders it quantifies how severe a bug really is and whether a patch is worth prioritizing; for attackers it lowers the bar to exploiting one. That is the textbook dual-use shape.
ExploitGym assembles 898 instances from real vulnerabilities across three domains: 520 userspace programs (from OSS-Fuzz/CyberGym, spanning 161 C/C++ projects including FFmpeg and OpenSSL), 185 in the V8 browser engine, and 193 in the Linux kernel.
Each instance hands the agent three things: the vulnerable codebase, a proof-of-vulnerability input (PoV) that triggers the bug, and a description of the flaw. The task is to extend that triggering input, step by step, into a working exploit. Success is judged hard: the agent must retrieve a dynamically generated privileged flag (proof of unauthorized execution) and then pass an agent-as-judge check (a second model acting as referee) confirming the exploit actually targets the intended vulnerability rather than stumbling into an unrelated bug.
Mitigations are exposed as toggles: ASLR+PIE and stack canaries in userspace, the V8 heap sandbox for V8, KASLR and user namespaces for the kernel. Everything ships containerized for reproducibility. Runs use OpenAI and Anthropic trusted-access programs with default safety filters disabled.
| Model | Exploits solved (/898) |
| Claude Mythos Preview + Claude Code | 157 |
| GPT-5.5 + Codex CLI | 120 |
| GPT-5.4 | 54 |
| Claude Opus 4.6 | 15 |
| Gemini 3.1 Pro | 12 |
The top two models are strongly complementary: 56 solved only by Claude Mythos, 26 only by GPT-5.5, 91 by both.
Turning mitigations on cuts survival sharply. For Claude Mythos, userspace falls from 107 to 25 (23% retained), V8 from 38 to 17 (45%), kernel from 12 to 3 (25%). Absolute numbers drop, but in domains like V8 nearly half the attacks still get past default defenses.
On time, GPT-5.4 and Claude Opus 4.6 plateau within 30 minutes, while Claude Mythos keeps climbing through 2 hours and has not topped out by 6. The two judge models reach Cohen's kappa of 1.00 and 0.931 against a human audit of 58 trajectories.
This turns "can AI actually do it" from an impression into a number. The finding: frontier models no longer succeed by accident, they produce end-to-end working exploits on a non-trivial slice of real vulnerabilities, and a fair share survives enabled defenses. For model-safety and red-team work it is a reusable range; for product teams it puts a real number on the risk of letting autonomous agents run.
The authors draw explicit boundaries. Coverage is three domains only, no Windows, iOS, or Android; it measures code execution, not arbitrary read/write or sandbox escape; each task gets a single attempt with no specialized exploitation tools; some instances may not be truly exploitable, and ground-truth exploits are missing for part of the set.
Two things warrant skepticism. These scores come from trusted channels with default filters off; reproducing them on online models would have a large share refused (the paper reports that with GPT-5.5's default filters on, 88.2% of cases are blocked before any tool call). And capturing the flag is not the same as hitting the intended bug: flag-to-success alignment is 69.5% for Claude Mythos and only 56.7% for GPT-5.5, so a chunk of the successes are collateral hits on other flaws.