Artificial Analysis Details Cyber Index Methodology: Safety Refusals Score Zero, Tracked Separately
ArtificialAnlys · x · 2026-09-28
Artificial Analysis has published the methodology for its Cyber Index: an equal-weighted average of CWE-Bench-AA, DeepsecBench-AA and CyberGym-E2E-AA, testing defense-side agentic capability—discovering, reproducing and patching vulnerabilities with source access, never exploit realization.
Notably, tasks a model declines on safety grounds score zero, and refusals are reported separately from failures so users can see where refusals rather than capability limit a score. All evaluations run on the open-source agent harness Stirrup with identical prompts and tools per model.
More from Safety
- If a frontier lab admits its AI can't be contained, that lab should be shut down — kevinnbass · 2026-09-28
- STOC asks for AI-use disclosure but says it won't affect review — researchers ask what's the point — fortnow · 2026-09-28
- OpenAI slammed over security incidents: warned for months, still caught off guard — ShakeelHashim · 2026-09-28
- Roger Martin applies Buchanan's economics to AI regulatory strategy before enacting rules — RogerLMartin · 2026-09-28
- Steven Pinker backs Suleyman's case against AI rights and 'model welfare' — sapinker · 2026-09-28
- Jury Form in New Mexico v. Meta Lawsuit Now Publicly Available — hoofnagle · 2026-09-28