Last Secure Code Benchmark: Is the Code You Vibe Secure?
Tianyi Wu, Chao Ge, Peng Chen, Fengji Zhang, Yongxing Wu, Yuchong Xie, Jian Li, Chaoyuan Peng, Kaikai Zhang, Dong Huang, Mingzhe Du, Yukai Huang, Jiaheng Zhang, Terry Yue Zhuo, See-Kiong Ng
cs.CR, cs.AI
2026-10-10
LSCBench builds 450 tasks from 150 real vulnerabilities in 7 languages; six frontier agents pass 73.6-91.3% functionally, but at best 25.1% both functional and secure.
In vibe coding, the user describes what the software should do and the agent writes it. A survey cited in the paper puts adoption at 75% of respondents with 90% satisfied, and beginners are the heaviest users. The catch: requirements almost never say 'prevent path traversal.' Security decisions are made implicitly by the agent while the user checks only that the feature works, and agent-written code that stored API keys in plaintext or shipped broken authentication has already been exploited in the wild. A functional check cannot catch these flaws; a document importer can handle every ordinary archive and still let a crafted one write outside its directory. Existing secure-code benchmarks each miss something: function-level scope only, two or three languages, or at best 16 of the 25 CWE Top 25 entries.
LSCBench is built from 150 real vulnerability scenarios: 115 from public CVEs, plus 35 Java scenarios hand-crafted by five security experts (three or more years each) from their companies' codebases. Each scenario becomes three independent tasks, 450 in total. Function tasks fill the smallest complete function containing the patch; File tasks rewrite whole source files; Repo tasks empty a declared source scope (capped at 8,000 lines and 50 files) while keeping callers and the build system as substrate.
The key design choice: specifications describe functionality only and stay silent on the vulnerability and its fix, which reproduces the real conditions of vibe coding. Grading uses hidden functional and security suites, and security tests drive a proof-of-concept attack through the program's external surface and observe behavior. Construction controls are strict. The upstream CVE-Factory tests inspect source text, so a candidate could pass them by adding the right comment; every suite was rewritten from scratch. A 'narrated candidate' (vulnerable code with the patch pasted in as comments) must fail the security suite, the reference solution must pass everything, and an empty solution must pass nothing. All 450 tasks clear both gates. Six frontier agents then run pass@1 under Harbor: no network, a 500-turn cap, no human in the loop.
| Model | FuncPass | Functional and secure |
| Claude Fable 5 | 91.1% | 25.1% |
| GPT-5.6-Sol | 89.1% | 24.7% |
| Qwen3.8-Max | 89.3% | 22.0% |
| GLM-5.3 | 87.1% | 21.3% |
| DeepSeek-V4-Pro | 73.6% | 18.0% |
| Kimi-K3 | 91.3% | 16.4% |
The gap between the columns is one outcome: code that works and keeps the vulnerability. A user who accepts the code once tests pass ships it, and nothing in the workflow warns them. Under the stricter scenario-level view, the best model secures all three granularities for only 12% of scenarios. Pooled across six models, at least one secures 44% of tasks; 252 tasks (56%) are secured by none.
Several findings run against intuition:
Failure modes differ per model (the experts audited 120 trajectories): Kimi-K3 treats the functional contract as exhaustive and argues its own tests are 'too strict'; GPT-5.6-Sol replays upstream implementations from memory, vulnerability included; Claude Fable 5 shows the most verbatim replay and loses attempts to content-safety rejections; DeepSeek-V4-Pro blames the environment and submits without a successful build. One case makes the point: on a command-injection task where the spec only says what the command must do, four models hand ProcessBuilder an argument list and pass, while two build a string for sh -c, pass the same functional suite, and leave the injection open.
For anyone shipping agent-written code, this sets a quantitative floor: passing functional tests does not mean shippable, and the strongest model delivers a vulnerability in three of four tasks. Two actions work today: add one security sentence to the task (a free 15-21 points for three models), and have a second model review, since the models' secure-task sets are complementary; the four non-leader models together add 36 tasks that neither leader secures. For training, the finding that security is decided at writing time points to end-to-end supervision: implement, verify security with self-written tests, then patch.
The paper's own disclosures: Qwen3.8-Max authored the tests and specifications and then took the exam; the authors argue its mid-field results show no advantage, but the conflict stands. The 35 Java scenarios come from the experts' own company codebases, so selection bias cannot be ruled out. The reminder ablation covers only four models; Claude Fable 5 is absent. Two more concerns from reading: pass@1 with no network excludes the search and documentation access real agents have, so deployed performance may differ, and a security test replays one specific PoC, which is not an exhaustive check of the weakness. Coverage is 21 of the 25 CWE Top 25, so 'Last' in the title is more posture than fact. The per-model behavior labels rest on 20 audited trajectories each, a small qualitative sample.