SRE Bench: GPT-6 scores 57% at binary reverse engineering, DeepSeek models under 2%
teortaxesTex · x · 2026-10-25
A Vals SRE Bench leaderboard discussion went viral: the benchmark tests whether AI agents can figure out what a real-world binary does without its source code. teortaxesTex highlighted that Chinese models, DeepSeek specifically, perform poorly at this task.
Key numbers:
- GPT-6 Astra leads at 56.87% accuracy, costing $10.24 per test
- GPT-6.1 Sol 50.76% (only $2.69/test) and Gemini 4 Argon 44.27% follow
- Claude Opus 5.5 scores 33.59%; Claude Sonnet 5.5 30.15%
- GLM 5.3 is the top Chinese model at just 9.16%
- DeepSeek V4 Pro 0813 scores 1.91% and DeepSeek V4.1 Flash 0.76%, near the bottom
- Cost per test ranges from $0.40 (MiMo V2.6 Flash) to $64.77 (Grok 4.7)
The author argues recent DeepSeek models should now reach at least 9% and pinged the benchmark maintainers for an update.
More from Models
- Dev: DeepSeek is very hard to beat on easy tasks, 10x cheaper than Sonnet — bindureddy · 2026-10-25
- OpenAI's rumored 6.1 Terra points to demand for smaller, cheaper models — haider1 · 2026-10-25
- DeepMind-backed benchmark: best AI agent solves 49/50 unpublished research-level physics problems — rohanpaul_ai · 2026-10-25
- MOSS-Image: 5B open image model hits 1.3M downloads in four days — neurocartel · 2026-10-25
- Why DeepSeek's Flash models dazzle but Pro versions disappoint: a training-signal theory — teortaxesTex · 2026-10-25
- Pro user: 62,500 Codex credits ($2,500) lock to smallest models the day your plan ends — lanqo88 · 2026-10-25