Deepsec benchmark scores model cost, speed, and accuracy on real security reviews
cramforce · x · 2026-07-28
Deepsec becomes a cybersecurity benchmark for real codebases
The post says the team turned deepsec into a cybersecurity eval and argues that it is useful because it measures a real-world task that consumes real time and money.
Key points from the quoted benchmark:
- It evaluates accuracy, cost, and speed for vulnerability finding.
- The cost data and quality tradeoffs map directly to production workloads.
- Running deepsec on larger codebases scales linearly in cost.
- The open-source tool review flow works on all existing code, not just pull requests, and supports scaled VM fanout and strong agent sandboxing.
The benchmark results referenced in the quote suggest a cost/performance tradeoff across models, with GPT-5.6 Sol leading on score, Kimi K3 delivering about half the top score at one-fifth the cost, and Grok 4.5 taking the best score/cost ratio among the top 10.
More from coding & agent
- OpenAI Codex appears to be filling a sessions folder with 360 GB of data — HankYeomans · 2026-07-28
- Pricing pages now ship as Markdown so AI agents can parse them — stuffyokodraws · 2026-07-28
- Agent search engine claims 62ms P50, about 6× faster than Exa — Scobleizer · 2026-07-28
- Kimi K3 lands in Cursor with US inference and zero data retention — togethercompute · 2026-07-28
- Local LLM agent drives Perfect Dark on Mac with 50 ms combat control — moyoteg · 2026-07-28
- Prompt caching can cut agent token costs by 50% to 90% on repeated requests — blaizedsouza · 2026-07-28