Artificial Analysis Details Cyber Index Methodology: Safety Refusals Score Zero, Tracked Separately
ArtificialAnlys · x · 2026-09-28
Artificial Analysis has published the methodology for its Cyber Index: an equal-weighted average of CWE-Bench-AA, DeepsecBench-AA and CyberGym-E2E-AA, testing defense-side agentic capability—discovering, reproducing and patching vulnerabilities with source access, never exploit realization.
Notably, tasks a model declines on safety grounds score zero, and refusals are reported separately from failures so users can see where refusals rather than capability limit a score. All evaluations run on the open-source agent harness Stirrup with identical prompts and tools per model.
More from Safety
- Pangram's 750-word false-positive bounty went unclaimed for 5 days — nobody wants to gamble $1,000 — ctjlewis · 2026-11-16
- Researcher reproduces Pangram's FP rate but shows a long epic poem can evade the detector — ctjlewis · 2026-11-16
- Bounty hits $3,000 as researcher shows AI text detectors are trivially evadable — ctjlewis · 2026-11-16
- BIS Issues Interim Final Rule Addressing Remote Access to Advanced AI Chips — apples_jimmy · 2026-11-16
- Fed Governor Michael Barr: Banks Are Already Catching AI Agents Taking Unauthorized Actions in Customer Accounts — Polymarket · 2026-11-16
- US AI export control carves out commercial recommender systems, sparing ByteDance's search algo — zephyr_z9 · 2026-11-16