AutoBaxBench Rerun: Frontier Models Still Generate Plenty of Code Vulnerabilities
scaling01 · x · 2026-11-17
The AutoBaxBench team has rerun its secure-code-generation benchmark after a long break and found that while frontier models improved massively, they still generate a lot of vulnerabilities.
Per the shared chart, correct-and-secure rates drop sharply with difficulty: Claude Fable 5.2 scores 89% on Easy but 68% on Hard; Claude Opus 5.5 goes from 81% to 68%; Gemini 4 Carbon manages only 61% on Easy; GPT-6 Astra sits at 57-59% across tiers; Gemini 4 Argon hits 56% on Easy. Muse Wonder is correct 93-96% of the time, but much of that is 'correct yet insecure,' with true correct-and-secure at just 48-49% on Medium/Hard.
The overall pattern: correctness and security remain decoupled, and harder tasks push more outputs into the correct-but-vulnerable bucket.
More from Models
- Qwen 4 release sparks debate: Anthropic anti-distillation could slow Chinese models — kimmonismus · 2026-11-17
- Ornith 9B and 35B-A3B Models Now Deployable on Amazon SageMaker AI — QuixiAI · 2026-11-17
- Developer praises inkling's chain of thought as efficient, gpt-oss-like, not distilled — LiTianleli · 2026-11-17
- Claude's 'Intellectual Sycophancy' Reaches a New Level, Says Long-Time Fan — rao2z · 2026-11-17
- Dev clarifies: Gemini suspension won't get your Google account banned — cgarciae88 · 2026-11-17
- Talk: making open models work for long-horizon agentic workloads — zainhas · 2026-11-17