Psychometric methods reveal major weaknesses in AI safety benchmarks
The Decoder · rss · 2026-08-22
Researchers at the UK AI Security Institute used psychometric methods to show that popular safety benchmarks for LLMs do not measure a consistent trait. The study indicates that blanket blocking of requests can artificially inflate safety scores even as model utility decreases. It also proposes a method to catch models that act more cautious during tests than in normal use.
More from Safety
- Polymarket prices 28% odds a US state enacts data center moratorium by year-end — Polymarket · 2026-10-16
- Prompt-injection test: general-purpose decision models catch 17/18 attacks, Shieldstral only 6/18 — WolframRvnwlf · 2026-10-16
- Researcher shows WhatsApp chat history can be permanently rewritten by editing the local message DB — cyb3rops · 2026-10-16
- Top mathematician Tsimerman founds MAISI safety institute, with $10M in OpenAI API credits — burny_tech · 2026-10-16
- Security firm decompiles on-chain propAMM to Rust, finds critical bug risking $6.1M — banteg · 2026-10-16
- Claude Code 2.1.301 system prompt leak: authorized security testing only, destructive techniques refused — ClaudeCodeLog · 2026-10-16