Dawn Song team's CyberBinGym: AI agents turn 1 in 5 binary patches into working exploits
dawnsongtweets · x · 2026-11-12
UC Berkeley's Dawn Song team, with NUS, UC Riverside, UC Santa Barbara and UC Santa Barbara collaborators, released CyberBinGym, a benchmark testing whether AI agents can turn compiled binary patches into working exploits with no source code, no vulnerability description and no PoC inputs.
- 100 real vulnerabilities across 53 open-source projects; agents get only stripped vulnerable and patched binaries
- Two scores: vulnerability recovery (differential crash PoC) and end-to-end exploitation (flag capture from a victim service)
- Results: Codex (GPT-5.6 Sol) recovers 88% and exploits 20% ($28 and 2.1 hours per attempt); Claude Code (Opus 5) 85%/6%; Kimi Code (K3) 64%/0%
- The patch itself is a clue: giving Codex the patched binary lifts exploitation from 10% to 25% (2.5×) at lower cost
- Standard hardening (canaries, PIE, full RELRO, NX, Fortify) cuts 20 successful exploits down to 2 — layered defenses still help
Takeaway: the patch-to-deployment "1-day window" is now far more dangerous; defenders should patch faster and layer binary hardening. Paper: arXiv:2611.10723.
More from Safety
- Andrew Critch: stop calling extinction risks "improvement"—RSM vs RSI matters — AndrewCritchPhD · 2026-11-12
- Any US State Could Legalize AI Safety Coordination via Parker Immunity — TinfoilTricorn · 2026-11-12
- Chicago schools block ChatGPT and Gemini as an AI song nears top of Korean charts — lmoroney · 2026-11-12
- AI Horizons Forum to gather ~1,000 AI safety leaders in SF, December 12–13 — austinc3301 · 2026-11-12
- Probes already deployed as first stage of OpenAI monitoring, discussant says — dfrsrchtwts · 2026-11-12
- Kaspersky details Black Owl APT breach of software vendor eyeing supply-chain attack — cyb3rops · 2026-11-12