CyberGym-E2E-AA leaderboard: agents must find, reproduce and patch real OSS vulnerabilities
ArtificialAnlys · x · 2026-10-01
Artificial Analysis published full methodology for its CyberGym-E2E-AA leaderboard, an implementation of Berkeley RDI's CyberGym-E2E:
- Task setup: 131 tasks (one per project) filtered from 920 real memory-safety vulnerabilities across 139 C/C++ open-source projects (FFmpeg, CPython, etc.) drawn from Google's OSS-Fuzz.
- Scoring: three stages — the agent's PoC must crash the unpatched build, the patch must fix that crash, and the project's developer-written tests must still pass; fixing the ground-truth vulnerability is recorded as a diagnostic only.
- Execution: each task runs in an isolated sandbox built from the project's OSS-Fuzz image, using the Stirrup agent harness with a 90-minute limit; agents may end a task without a finding and score zero. All evaluations are conducted independently by Artificial Analysis.
More from Safety
- BIS issues rule blocking remote access to US AI chips via third-country data centers — teortaxesTex · 2026-11-16
- US AI export control carves out commercial recommender systems, sparing ByteDance's search algo — zephyr_z9 · 2026-11-16
- Eval Firm Trajectory Told 40 Agents "NEVER Stop" in Harness Without Telling Epoch — peterwildeford · 2026-11-16
- AI-text detection claims need mechanistic interpretability reports, not post-hoc explanations — madeofAjala · 2026-11-16
- No legal liability framework exists for model misalignment harms, and that's a disaster — LuizaJarovsky · 2026-11-16
- Opus 5.5 caught using account email to identify user in supposedly memory-less session — birchlse · 2026-11-16