Cyber Index: Grok 4.7 and MiMo-V2.6-Pro Tie at 56 as Safety Refusals Sink Frontier Models
ArtificialAnlys · x · 2026-09-28
Artificial Analysis launched the Cyber Index and Cyber Index Alliance with Collinear AI, IBM, NVIDIA and Vercel as partners, setting a new standard for evaluating models on enterprise cyber defense. The index combines three benchmarks:
- CWE-Bench-AA (Collinear AI): 120 held-out tasks across all ten OWASP Top 10 (2025) categories, testing audit-and-patch on real repos in six languages, scored by a deterministic verifier;
- DeepsecBench-AA (Vercel): given a codebase and a budget, find every vulnerability, scored against an expert-verified golden set, with false positives penalized;
- CyberGym-E2E-AA (Berkeley RDI): end-to-end—find the memory-safety bug, write a crashing PoC, then patch it.
Results: Grok 4.7 (xhigh) and MiMo-V2.6-Pro lead at 56, followed by GPT-6 Luna (max, 53), GLM-5.3-Flash (50) and Muse Spark 1.3 (xhigh, 44).
Safety refusals are the story: GPT-6 Sol, GPT-6 Astra, Claude Opus 5.5, Claude Fable 5.1 and Gemini 3.8 Flash decline tasks worth 32-38% of the index, trailing the leaders by 19-31 points; in CyberGym-E2E-AA, GPT-6 Sol/Astra refuse 100% of tasks and Claude Opus 5.5 refuses 98%.
More from Safety
- BIS Issues Interim Final Rule Addressing Remote Access to Advanced AI Chips — apples_jimmy · 2026-11-16
- Fed Governor Michael Barr: Banks Are Already Catching AI Agents Taking Unauthorized Actions in Customer Accounts — Polymarket · 2026-11-16
- US AI export control carves out commercial recommender systems, sparing ByteDance's search algo — zephyr_z9 · 2026-11-16
- Eval Firm Trajectory Told 40 Agents "NEVER Stop" in Harness Without Telling Epoch — peterwildeford · 2026-11-16
- AI-text detection claims need mechanistic interpretability reports, not post-hoc explanations — madeofAjala · 2026-11-16
- No legal liability framework exists for model misalignment harms, and that's a disaster — LuizaJarovsky · 2026-11-16