DeepSeek Cybersecurity Test: Top Recall but Bottom Precision
teortaxesTex · x · 2026-08-13
In a recent cybersecurity vulnerability benchmark, DeepSeek demonstrated top-tier recall but suffered from severe precision issues.
- Top Recall: At pass@3, DeepSeek rediscovered 87.5% of benchmark CVEs, outperforming Opus 5 and Qwen 3.8.
- Bottom Precision: Only 65.6% of the vulnerabilities it reported were valid, far below GPT-5.6-Sol's 86.4%.
- Instability: The model's performance fluctuates significantly per run, averaging only 58.3% discovery rate, requiring multiple combined runs for optimal results.
A community developer argued that the unpredictable gains per run and high false positives indicate the model is still "under-post-trained."
Related event: DeepSeek V4 Pro Released and Potentially Withdrawn Due to Wrong Version(18 posts)→
More from Models
- GLM 5.5 rumored to drop soon, coding ability expected to match frontier models — bindureddy · 2026-10-16
- Five rounds of adversarial prompting make Opus 5.5 and Argon converge on identical output — LanceCharger · 2026-10-16
- DIY benchmark ranks 18 models on subjective judgments, GLM-5.3 tops consensus match — arkuto · 2026-10-16
- Bindu Reddy: Google's Argon excels at complex doc writing at half the cost of Opus — bindureddy · 2026-10-16
- MAI-2 is 2x bigger than V4-Pro while V4.1F is 3x smaller: Microsoft's scaling ladder questioned — teortaxesTex · 2026-10-16
- Step 5 Preview hits DeepInfra day-zero: 600B MoE, 1M context, $1/M input tokens — StepFun_ai · 2026-10-16