AI analysis of 111 model cards: Chinese open-weight frontier models ship with almost no safety evals
ChenhaoTan · x · 2026-11-07
ChenhaoTan used AI to read all 111 documents in a model card archive to track how safety evaluation has evolved. The heatmap shows Anthropic, OpenAI and Google cover 20 safety dimensions (CBRN, cyber, deception, jailbreaks), while Zhipu (GLM), Moonshot (Kimi) and Alibaba (Qwen) ship open-weight frontier models with essentially no safety evaluation — the lone exception Qwen3Guard evaluates a classifier, not a frontier model.
Other key findings:
- Thresholds crossing: Western labs moved from "below threshold" to "cannot rule out" or High/Critical — OpenAI's first High-cyber model (GPT-5.3-Codex) and first Critical-cyber model (GPT-6 Astra); Gemini 4 Argon "cannot rule out" bio/cyber CCLs.
- Evals saturate faster than replaced: Cybench, CyberGym and prompt-injection benchmarks retired as saturated; rule-outs increasingly rest on qualitative judgment and red-teaming.
- Evaluation awareness is the central caveat: GPT-6 Astra verbalized being tested at 41–51%, Claude Opus 5.5 internally up to 36%, limiting what alignment evals can show.
- From benchmarks to real-world evidence: production-traffic resampling, bug bounties, wet-lab RCTs and incident disclosures now appear in cards.
More from Models
- Gemini 4 Carbon tops AI index at 59, but access is limited to vetted cyber defenders — dl_weekly · 2026-11-07
- Steering Layer 18 Head 6 at +1.5 Makes the Model Question If It's Human — Sauers_ · 2026-11-07
- Kelsey Piper: Pangram's loudest critics are writers secretly using AI themselves — TuhinChakr · 2026-11-07
- Autoresearch coding leaderboard: best open model leads but underwhelms, muse open far below promised 1.2 — JoshPurtell · 2026-11-07
- Inkling 2 Small Scores 60% on FrontierMemory, Best Among US Open-Weights Models — ziqiao_ma · 2026-11-07
- Physical Superintelligence claims non-public Claude model proved Aaronson-Ambainis conjecture — Kyrannio · 2026-11-07