DeepsecBench vulnerability-finding eval: GPT-6 Sol leads at 46%
ArtificialAnlys · x · 2026-09-28
Artificial Analysis highlights DeepsecBench-AA (from Vercel), which isolates vulnerability discovery: given a codebase and a budget, an agent must find every vulnerability, scored by F2 against an expert-verified golden set, weighting recall over precision.
- GPT-6 Sol (max) leads at 46%, ahead of GPT-6 Astra (37%), DeepSeek V4.1 Flash (31%), Grok 4.7 (27%) and Claude Opus 5.5 (27%)
- The Score-vs-Cost Pareto frontier: GPT-6 Sol at $1.83/task, DeepSeek V4.1 Flash at $0.31/task, MiMo-V2.6-Pro at $0.09/task
More from Models
- Tencent Hy4 flash and InternLM Sage-S2.1-Flash both released, tool-calling tests requested — MajesticPuzzler · 2026-11-16
- User Shares Coral Reef Creation Labeled 'Qwen 4 Max (xhigh)', an Unannounced Model Name — intellectronica · 2026-11-16
- Stealth Model 'silicon-2' Surfaces in Devin, Suspected to Be Anthropic's Next Model — kimmonismus · 2026-11-16
- Google hides Gemini model label in AI Mode, rolls Gemini 4 Carbon to Pro subscribers — gaganghotra_ · 2026-11-16
- Open-weight Karpo-Decision-31B tops Decision Index at 73.9, one forward pass per answer — CodeByPoonam · 2026-11-16
- AI daily: DeepSeek rate-limits chats, Qwen haggling spurs shop warnings, Doubao workspace launch — 快鲤鱼 · 2026-11-16