Dawn Song's Team Releases CyberBinGym: AI Can Exploit Real Vulnerabilities from Binary Patches Alone

A UC Berkeley team led by Dawn Song, with NUS, UC Riverside and UC Santa Barbara, released a new benchmark, CyberBinGym, testing whether AI agents can turn a security patch into a working exploit using only compiled, stripped binary patches (no source code, no vulnerability description, no PoC). Tests show frontier agents can already do this: Codex reproduced 88% of vulnerabilities and achieved end-to-end exploitation in 20%, leading the authors to warn that the window between patch release and deployment is becoming more dangerous than ever.

Confirmed

• Benchmark setup: tasks cover 100 real vulnerabilities across 53 open-source projects, spanning six vulnerability families and three difficulty tiers; agents receive only the vulnerable and patched stripped binaries.
• Scoring design: "vulnerability reproduction" (PoC triggers a sanitizer failure on the vulnerable version but not the patched one) is scored separately from "end-to-end exploitation" (stealing a secret from an independent victim service), on the grounds that a crash doesn't equal exploitability.
• Scorecard: Codex (GPT-5.6 Sol) reproduced 88% and exploited 20%; Claude Code (Claude Opus 5) 85%/6%; Kimi Code (Kimi K3) exploited 0.
• Patches as clues: on 20 matched tasks, giving Codex the patched binary raised exploit success from 10% to 25% (2.5x) and lowered average per-task cost — even compiled, the patch diff leaks information.
• Mitigation: after recompiling the 20 targets Codex successfully exploited with standard binary hardening (stack canaries, PIE, full RELRO, NX, Fortify), successful exploits dropped from 20 to 2.
• The paper is on arXiv, number 2611.10723.

Why it matters

• Following CyberGym and ExploitGym, the team continues building yardsticks for frontier AI's real-world cyberattack capabilities; ExploitGym previously also inadvertently surfaced an OpenAI–Hugging Face-related incident (the post mentions model-level risk).
• The results show the patch itself becomes an attack clue: the authors recommend deploying patches quickly and layering defenses that still work before updates land (such as binary hardening), since standard hardening alone cut AI's successful exploits from 20 to 2.

2026-11-12 ~ 2026-11-12 · 9 related posts

• [source] Dawn Song team's CyberBinGym: AI agents turn 1 in 5 binary patches into working exploits — dawnsongtweets · 2026-11-12
• From CyberGym to CyberBinGym: scaling AI cyber benchmarks after the OpenAI–HF incident — dawnsongtweets · 2026-11-12
• CyberBinGym setup: 100 real CVEs, 53 projects, stripped binaries only — dawnsongtweets · 2026-11-12
• A crash is not an exploit: why CyberBinGym scores recovery and exploitation separately — dawnsongtweets · 2026-11-12
• [source] CyberBinGym scores: Codex 88% recovery / 20% exploit, Kimi Code 0% exploitation — dawnsongtweets · 2026-11-12
• Patches are clues: giving Codex the patched binary lifts exploit rate from 10% to 25% — dawnsongtweets · 2026-11-12
• Standard binary hardening cuts Codex's 20 AI exploits down to just 2 — dawnsongtweets · 2026-11-12
• The patch-to-deployment window matters more than ever, CyberBinGym authors warn — dawnsongtweets · 2026-11-12
• [source] CyberBinGym paper live on arXiv: Dawn Song team's binary-exploit benchmark — dawnsongtweets · 2026-11-12