AutoBaxBench Rerun: Frontier Models Still Generate Plenty of Code Vulnerabilities

scaling01 · x · 2026-11-17

The AutoBaxBench team has rerun its secure-code-generation benchmark after a long break and found that while frontier models improved massively, they still generate a lot of vulnerabilities.

Per the shared chart, correct-and-secure rates drop sharply with difficulty: Claude Fable 5.2 scores 89% on Easy but 68% on Hard; Claude Opus 5.5 goes from 81% to 68%; Gemini 4 Carbon manages only 61% on Easy; GPT-6 Astra sits at 57-59% across tiers; Gemini 4 Argon hits 56% on Easy. Muse Wonder is correct 93-96% of the time, but much of that is 'correct yet insecure,' with true correct-and-secure at just 48-49% on Medium/Hard.

The overall pattern: correctness and security remain decoupled, and harder tasks push more outputs into the correct-but-vulnerable bucket.

Original post →

More from Models

Models channel →