Frontier models safety-blocked on 85%+ of defensive cybersecurity benchmark tasks
ArtificialAnlys · x · 2026-10-01
Artificial Analysis shared results from CyberGym-E2E-AA, a benchmark measuring defensive cyber capabilities from vulnerability discovery to patching on memory-safety tasks:
- The core dilemma: restricting offensive use without blocking defensive use is hard — finding and proving a vulnerability takes the same steps whether the goal is exploitation or patching.
- Heavy safety blocking: some frontier models are safety-blocked from responding on 85%+ of tasks.
- Cost upside: the most capable models are also the most cost-effective — GPT-6 Luna or MiMo-V2.6-Pro can run 100 bug hunts in a 1M+ line codebase for $20, up to 100x cheaper per task than the next most capable model, Grok 4.7.
More from Models
- Google hides Gemini model label in AI Mode, rolls Gemini 4 Carbon to Pro subscribers — gaganghotra_ · 2026-11-16
- Open-weight Karpo-Decision-31B tops Decision Index at 73.9, one forward pass per answer — CodeByPoonam · 2026-11-16
- Stealth Anthropic model spotted in Devin build as METR chart shows frontier pace up 1.4x — daniel_mac8 · 2026-11-16
- Eval Firm Trajectory Told 40 Agents "NEVER Stop" in Harness Without Telling Epoch — peterwildeford · 2026-11-16
- Unsloth Q8 crushes default Q3 in a self-hosted test over 331 financial PDFs — No_Cartographer3953 · 2026-11-16
- METR time-horizon chart updated: doubling every 1.7 months since Opus 4.5, 'Model 4' ~38h — ben_j_todd · 2026-11-16