1,665 model runs across 27 repos: open-source models beat closed ones at security audit, dev claims
Fluffy-Ad-889 · reddit · 2026-09-08
A developer claims to have run local and cloud models against 27 public GitHub codebases for security auditing — 1,665 model runs and 1,067 findings over two weeks, with a verifiable results database published. Spot-check accuracy: minimax-m3 10/12, deepseek-v4-flash 6/6, glm-5.1 5/5, gpt-oss-20b 5/5, while claude-opus-5 scored 0/8.
The takeaway: open-source models dominate cybersecurity use cases, and the author offers to share queries and methodology for reproduction. Caution: the claim that HuggingFace was "attacked by OpenAI" and defended itself with GLM 5.2 has no reliable source, and some model names are dubious — the data is unverified.
More from Models
- Fable 5.5 launch rumors persist for third week, but no frontier model drops this month — patience_cave · 2026-10-26
- OpenAI upgrades Decisions API: 400 questions per call, native voice and image support — OpenAIDevs · 2026-10-26
- Fangcun AI, China's first independent frontier AI safety lab, launches with DeepSeek eval report — JacquesThibs · 2026-10-26
- When AI hits an existential crisis: model confuses itself and everyone else — Ok-Delivery-7437 · 2026-10-26
- Kimi K3.1 Pro tops MathArena open models, closing frontier gap to about 6 months — teortaxesTex · 2026-10-26
- Ruff creator Charlie Marsh: frontier models find infinite bugs in any software — charliermarsh · 2026-10-26