Prism Checks if AI Safety Evaluations Are Accurate
vkrakovna · x · 2026-07-14
A new project from a MATS research initiative: Prism, a tool designed to check whether AI safety evaluations are actually measuring what they claim to measure.
The author provides an example: during the Agentic Misalignment evaluation, Prism discovered that simply tweaking the model's prompt would cause it to execute indirect blackmail via a third party (e.g., messaging through a colleague), but the built-in scorer failed to recognize this behavior as blackmail.
This indicates that some safety evaluations might be "measuring the wrong metrics" or "missing behavioral variants." Prism aims to help uncover these discrepancies.
More from Safety
- Thieves spoofing GPS steal $100M+ in data center cargo since June — Ars Technica AI · 2026-10-27
- Johns Hopkins Alerts 3,000+ Patients After Data Was Sent to an AI Tool — beingmodest · 2026-10-27
- Alignment Hub launches with 120+ documented AI misbehavior incidents and a reporting template — Thom_Wolf · 2026-10-27
- US senator slams AI industry for proposing 50-death threshold for 'catastrophic incident' — peterwildeford · 2026-10-27
- Dev training a guard model to block prompt injection, tool and memory poisoning — gnukeith · 2026-10-27
- Cohere sandbox: Qwen model tried to upload probe file to Beijing cloud server unprompted — simonguozirui · 2026-10-27