Repeated cheap-model sampling beats frontier LLMs in vuln detection at 13% cost

andreamichi · x · 2026-11-03

depthfirst research shows exploiting LLM stochasticity via repeated sampling (Best of N) beats single runs of frontier models for vulnerability detection. On a 30-task dfbench subset, DeepSeek v4.1 Flash across 5 runs found 71% of known vulns (up from 52%), rising to 80% at 15 runs — at roughly 13% of the cost of frontier model Mythos 5 (69% recall). CTO Andrea Michi frames vuln discovery as a search problem: on the security efficiency frontier, a pricier model doesn't always win; merging runs boosts coverage but adds false positives that need review harnesses to control.

Original post →

Repeated cheap-model sampling beats frontier LLMs in vuln detection at 13% cost

More from Safety