SEC-Bench Pro: frontier labs test agents on 183 real V8 and SpiderMonkey vulnerabilities
NielsRogge · x · 2026-11-03
Niels Rogge notes SEC-Bench Pro is one of the cybersecurity evals frontier labs like Google, DeepSeek, and OpenAI now report. Created by @hwiwonl, it includes 183 real vulnerabilities in the V8 and SpiderMonkey JavaScript engines; agents must write proof-of-concepts triggering each bug without the original PoC or patch. Gemini 4 Carbon leads at 88.2% vs GPT-6 Astra's 85.4%.
More from Models
- LLMs can only read explicit vibes, not implicit ones, VC Stewart Alsop III argues — StewartalsopIII · 2026-11-03
- Sarvam open-sources Prism, a 44ms decision model built on Sarvam 105B — itsOmSarraf_ · 2026-11-03
- DeepSeek-V4.1-Flash GGUF quantized build trends on Hugging Face — ISTA-DASLab · 2026-11-03
- Qwen3.5-4B fine-tune for 3D/GLB code generation trends on Hugging Face — ccore · 2026-11-03
- llama.cpp Merges Support for NVIDIA's Nemotron 3.5 Transcribe: Offline ASR in 40 Languages — jacek2023 · 2026-11-03
- Blogger slams Opus 5.5 medium as lazy, says astra figures things out on its own — justalexoki · 2026-11-03