Prism Checks if AI Safety Evaluations Are Accurate
vkrakovna · x · 2026-07-14
A new project from a MATS research initiative: Prism, a tool designed to check whether AI safety evaluations are actually measuring what they claim to measure.
The author provides an example: during the Agentic Misalignment evaluation, Prism discovered that simply tweaking the model's prompt would cause it to execute indirect blackmail via a third party (e.g., messaging through a colleague), but the built-in scorer failed to recognize this behavior as blackmail.
This indicates that some safety evaluations might be "measuring the wrong metrics" or "missing behavioral variants." Prism aims to help uncover these discrepancies.
More from Safety
- Study: AI reviewing AI misses most reward hacking; best models miss 25%+ of hacks — ronbodkin · 2026-10-26
- Fangcun AI, China's first independent frontier AI safety lab, launches with DeepSeek eval report — JacquesThibs · 2026-10-26
- ESR on the Bugpocalypse: locking down frontier models only hands the game to Chinese open weights — colinmcnamara · 2026-10-26
- OpenCode breach exposed AI API keys of 1.4% of Console users — thione · 2026-10-26
- DHS orders DeepSeek products removed from US civilian agency systems — thione · 2026-10-26
- One hacker breached 100+ company websites using AI, scope unknown — thione · 2026-10-26