Aikido benchmark: DeepSeek V4 Pro tops cyber security AI, open-source beats frontier
sull · x · 2026-08-23
Aikido released a report detailing a benchmark of 10 AI models' cyber capabilities, burning 11.7 billion tokens across 32 fresh vulnerabilities with three attempts each.
Key Findings:
- Top Performer: DeepSeek V4 Pro 0813 found the most vulnerabilities. Pooling three runs reaches 28 of 32.
- Cost Efficiency: The most expensive models are not required. Three DeepSeek Pro runs cost $295 and outperform Opus 5 or Grok 4.6. Three Flash runs cost $108, matching Grok's best pass for less than a quarter of the cost.
- Open-source Surge: Open-source models now outperform public frontiers in pooled vulnerability recall. DeepSeek V4 Pro topped all tested public closed models, followed by Qwen, Kimi, and GLM-5.3 with strong consistency.
- The Trade-off: While matching frontier performance at cheaper rates, open models produced the most false leads.
More from Models
- Gemini 4 Argon scores 97.5% on ARC-AGI-3, second lab above 95% — jasonbaldridge · 2026-10-16
- Liquid AI launches d1-ultra, topping Artificial Analysis Decision Index at 65 — JosephJacks_ · 2026-10-16
- Liquid AI's d1-ultra decides instead of writing, charging input-only $0.042 per million tokens — JosephJacks_ · 2026-10-16
- Meta open-sources Muse Spark, first base model from its new MSL org — RylanSchaeffer · 2026-10-16
- Meta open-sources Muse Spark Open: 816B MoE with 42B active, 1M context, strong benchmarks — testingcatalog · 2026-10-16
- Study: 87% of measured AI gains came by 2022, labs cherry-pick evals — gerardsans · 2026-10-16