CyberArena: live attack-defense benchmark drops agent capture rates to 35–52%
jiqizhixin · x · 2026-11-21
Researchers from 2077AI, UC Berkeley, UC Santa Cruz and UCSB introduce CyberArena, an arXiv benchmark where two cybersecurity agents attack and defend concurrently in isolated sandboxes—each must exploit the opposing service while patching its own and keeping it functional, with flags rotating each round.
- Scale: 12 model-and-harness combinations, 40 challenges, 466 matches; a dynamical scoring model estimates offensive and defensive ability from live match telemetry.
- Key result: on 30 challenges all three evaluated systems solved in solo mode, third-round capture rates fall to 52%, 37% and 35% under live opposition.
The takeaway: static Jeopardy-style evals overstate real-world security agent capability; active defense materially changes measured success.
More from Safety
- Semi-autonomous agents exploit CVEs at a pace 2005-era security norms can't handle — chrisrohlf · 2026-11-21
- Baseten breaks down its security threat model as agent swarms accelerate attacks — baseten · 2026-11-21
- Leaked call: green groups plotted to kill AI bill "under the guise of making it better" — aronchick · 2026-11-21
- Rumor: Anthropic's model may have cracked popular encryption algorithms, disclosed only to authorities — basedjensen · 2026-11-21
- OpenAI expands Daybreak security model access for defenders and opens GPT-6 Astra Blue alpha — OpenAIDevs · 2026-11-21
- AI regulation crux: skeptics ask how rules could be abused, not what risks they fix — herbiebradley · 2026-11-21