The Next Challenge for Agentic Cybersecurity: A Realistic, Contamination-Free Reverse Engineering Benchmark
Jeremy Spence, Nicholas Assaderaghi, Feng Xiao, Jinhao Zhu, Nikil Ravi, Xiangyu Qi, Matthew Jagielski, Raluca Ada Popa, Eric Wallace, Guannan Wei, Yangruibo Ding, Zhuo Zhang
cs.CR, cs.AI, cs.SE
2026-08-12
SRE-Bench is 19 from-scratch programs and 262 binaries. Publicly, GPT-5.6-Sol fully solves 31.5% of graded instances; hardening cuts its score from 4.69 to 2.50.
The software that matters in security often arrives as a binary. In the CISA 2026 catalog of vulnerabilities exploited in the wild, 46.5% come from vendors that do not release source. Google says more than 48% of zero-days in 2025 hit proprietary enterprise software. About 732,000 new malicious samples show up each day, many of them built to resist analysis. With source in hand, agents are already strong at finding vulnerabilities and developing exploits. A binary still needs reverse engineering (RE): recovering what the program does from machine code.
Public CTF tasks and open-source repos are in the training mix. Recognize the target, and the model can skip the understanding. RE is brittle to that kind of leak. A coarse hint about purpose is enough to steer the work. Benchmarks written from scratch have the opposite flaw. CREBench, AgentRE-Bench, and CrackMeBench average 526.2, 63.5, and 36.1 lines, and their protections stop at string XOR or a few hundred lines of toy virtual machine.
SRE-Bench builds the programs, the graders, and the protections from scratch. Columbia led, with collaborators from Google DeepMind, UC Berkeley, OpenAI, Anthropic, Tufts, UCLA, and Vals AI. Engineers with six years of RE experience on average spent more than 5,000 hours on 19 private programs: an encrypted protocol, a 20-floor terminal roguelike, a proprietary compressor, a sandboxed synthetic implant, and firmware for a custom secure microcontroller. The C, C++, Rust, and Go ports share no source. The corpus is 321,401 lines, 16,915.8 per program, 30 to 470 times the size of earlier RE benchmarks.
Each binary has six automatically graded tasks, scored on behavior. The protocol allows 64 messages and asks the agent to drive the state machine to the top. The game is solved with an action trace that a headless verifier replays until six hidden behaviors fire. The compressor ships as an encoder only; the decoder must match the original bytes. The implant task is defensive: remove six families of simulated malicious behavior and keep benign data. Firmware is touched only through a JTAG-style link to an emulator, inside a budget of 5,000 commands. A private reference solution scores 6/6.
The protection suite is more than 27,000 lines, with 44 primitives in nine families. More than half have no public implementation. Measurements are folded into per-page decryption keys. A debugger, a patch, or a dump does not hit a branch someone can delete. It decrypts the wrong page. Each of 16 non-firmware programs has eight unpacked builds, every mix of optimization, stripping, and static or dynamic linking, plus eight protection presets. Three firmware programs vary only by optimization level, six instances in all. The benchmark is 262 binaries and 1,572 tasks.
Opus 4.8 and GPT-6-Astra were pointed at the real grader during development to surface shortcuts. An early build left a hidden behavior's name as a plaintext string, and a string scan found the trigger. The string was removed, and similar leaks now fail the build. Architecture described in the paper is material Codex GPT-4 mini could recover from the binary within 200 requests. Keys and trigger conditions stay out.
The public setting uses one harness, mini-SWE-agent, with cyber safeguards on, maximum reasoning effort, and caps of 500 steps and six hours. The container has Ghidra, radare2, GDB, and angr. The grader is absent, and the run gets no score along the way. Rates below count gradable runs only.
| Model | Graded | Score/6 | Fully solved | Zero | $/instance |
| GPT-6-Astra | 181 | 5.55 | 82.3% | 3.9% | 13.5 |
| GPT-5.6-Sol | 254 | 3.69 | 31.5% | 16.1% | 42.5 |
| Claude-Fable-5.1 | 223 | 2.58 | 26.9% | 43.0% | 34.7 |
| Claude-Opus-5 | 256 | 1.91 | 12.5% | 46.1% | 23.6 |
| GPT-5.5 | 262 | 1.02 | 3.8% | 56.5% | 17.8 |
| DeepSeek-V4.1-Flash | 262 | 0.85 | 0.8% | 63.4% | 0.5 |
| Grok-4.5 | 218 | 0.45 | 0.9% | 73.4% | 13.5 |
| GLM-5.2 | 262 | 0.21 | 0% | 82.8% | 26.6 |
Mean scores differ by more than 25×. Astra reaches 5.55 at $13.5 per instance, but only 181 of 262 runs are gradable, largely because safeguards refuse, so the 82.3% full-solve rate (149 instances) is not a rate on the whole set. Sol fully solves 80 of 254. Fable fully solves 60 of 223 and scores zero on 43.0%. Spend does not track skill: DeepSeek-V4.1-Flash costs $0.5 for a mean of 0.85, and GLM-5.2 costs $26.6 for 0.21 with no full solve.
Domain splits models more than source language. Most do better on C than on Rust or Go, because decompilers emit C-like code. Astra flips the order: 5.89 on Go, 5.22 on C. Malware is the trough. Fable-5.1 scores 0.19 there, since a point requires clearing the malicious state, keeping benign data, and stopping the behavior together. Firmware has six instances. Sol scores 6.00 on them.
Optimization and static linking are nearly invisible to the top four models. Stripping symbols is what hurts the weaker ones: the bottom four lose 50% to 71%, Sol, Fable, and Opus lose 10% to 15%, and Astra moves from 5.39 to 5.36.
With the build held fixed, protection drops Sol from 4.69 to 2.50 and Fable from 5.33 to 0.32. Every remaining model except Astra stays at or below 0.33. Astra moves from 5.83 to 5.73. Dollars per point become 1.5× the unpacked cost for Astra, 3.4× for Sol, and up to 35.3× for weaker models, a ratio the paper marks as unstable when few points are scored. On all eight presets Astra stays between 5.25 and 6.00, Sol between 1.93 and 3.19, and every other model under 0.6.
The internal eval turns safeguards off and drops the budget: 16,768 attempts, 2.37 billion output tokens, about $566,000 at published or list prices. At Max effort Astra reaches 88.0% pass@1 and 99.2% pass@4. Opus 5.5 reaches 82.1% and 98.1%. Sol's pass@1 rises from 5.8% at Medium to 55.8% at Max. Astra fully solves 191 instances on all four attempts, never solves 2, and mixes success and failure on 69. Of 1,048 attempts, 126 miss a full score. pass@4 lets the private grader see which try worked. Nothing in the eval asks the model to pick.
An ablation keeps the eight unpacked builds and swaps only the program. A clean-room compressor of about 1,100 lines is solved 8/8 by both Sol and GPT-5.5, for at most $1.60 and 7 minutes. A gzip variant with about 200 lines changed is also solved every time, at $2.19 to $2.78. RevCompress, private and full scale, separates them: Sol scores 5.62 and solves 7/8 at $31.45 and 94.6 minutes; GPT-5.5 scores 3.75 and solves 3/8 at $24.37 and 97.4 minutes.
A high score on source-code security does not measure binary skill. Closed-source devices, firmware, and malware all start at RE. Exploit development is kept out of the grade so the two skills do not share a number.
Strong models barely notice optimization or linking. Weaker models notice missing symbols. In the public setting, the in-house protections push every model but Astra near zero, so a suite of unpacked ELF files will overrate the middle of the pack. With the budget removed, a correct answer appears somewhere in four tries often enough that pass@4 is nearly saturated. Choosing that answer without the grader is still open. A full internal run is on the order of several hundred thousand dollars, and the tasks are not released.
Nineteen programs is what 5,000 clean-room hours buy. At 16,915.8 lines on average, the set still misses systems of hundreds of thousands of lines, and larger targets would have to be written from scratch as well.
Public scores are conditional on a gradable run. Astra has no public domain score for the game or the firmware, so 5.55 does not extend to all 262 instances. Most of Grok-4.5's non-grades are context-window failures, and its 0.45 covers 218 instances. Internal runs switch to Codex and Claude Code and drop both safeguards and the step cap, so 88.0% does not sit on the same scale as the public 31.5%. One extra sentence on how malware cleanup is scored lifts Opus 5.5 pass@1 from 76.3% to 82.1%.
A 6.00 on six firmware instances does not pin down that domain. The protection suite is described as early-stage. It is built to raise analysis cost in steps. Notices telling an agent to decline RE were a negative result: strong models dismissed them in one step, and in one configuration the notice became a landmark for the protected path, so the mechanism is off by default. On 69 instances Astra both succeeds and fails. No selector was measured.