Study of 135 model releases: AI labs barely share safety benchmarks, top one used by only half
ChenhaoTan · x · 2026-11-13
Inspired by Jason Wei, SAIL Lab analyzed the model cards, system cards and safety reports behind 135 releases from 18 labs in 2026 — including OpenAI, Anthropic, Google DeepMind, Meta, xAI, DeepSeek and Alibaba Qwen — to benchmark safety benchmarks themselves. Key findings: unlike capability benchmarks, labs share very few safety benchmarks (the top one is reported by only half of the labs); most safety results are hard to verify externally; and the large reporting gap between labs needs to be addressed. Chenhao Tan highlights UC Berkeley EECS and Dawn Song as a powerhouse in this space.
More from Safety
- Student's AI agent defended his school's network from a cyberattack after he left for college — heyneighbor · 2026-11-13
- Selling AI traces has reputable buyers, but Arca isn't one, says dev — jimmykoppel · 2026-11-13
- Dev exposes Arca, an app paying for your discarded vibe-coding traces, as a crypto-backed scam — jimmykoppel · 2026-11-13
- AI voice cloning can sound like family — and the best defense is surprisingly simple — petrusenko_max · 2026-11-13
- Brave v1.98 strips hidden FBMD identifiers Meta embeds in your photos — gnukeith · 2026-11-13
- DC Council fails to pass autonomous vehicle bill, stalling Waymo deployment — Scobleizer · 2026-11-13