CWE-Bench, a frontier cyber benchmark where agents find and fix real vulnerabilities, goes open-source
mmitchell_ai · x · 2026-11-04
Collinear AI and nazneenrajani's team open-sourced CWE-Bench on Hugging Face, calling it the frontier cyber benchmark and the largest cyber release in the AI community.
- Each task gives an agent a real open-source repository and an audit instruction
- The agent must find and fix the vulnerability without being told what or where it is
- Deterministic verifiers check that the fix blocks the exploit while preserving legitimate behavior
- The team argues the community should be able to scrutinize and contribute to such impactful benchmarks, and welcomes feedback
More from Safety
- AI safety researcher Jeff Ladish warns RSI is nearing and scarier than any mountain lion — JeffLadish · 2026-11-04
- Google AI division workers push execs to waive non-disparagement agreements to warn public about AI risks — EthanJPerez · 2026-11-04
- 14 Agentic AI Security Projects Every AI Security Engineer Should Build — Full-Presence7590 · 2026-11-04
- AI welfare and AI safety are allies, not rivals — a researcher's four arguments — RosieCampbell · 2026-11-04
- Meta's letter to Newsom opposes mandatory outside safety checkers, diverging from OpenAI and Anthropic — ShakeelHashim · 2026-11-04
- New book "Treacherous Tech" cites Irving's call to halt frontier AI and Pachocki's warnings — DavidSKrueger · 2026-11-04