Emergent Misalignment is predictable before training via activation-space distance, new study shows
ChenhaoTan · x · 2026-09-04
New research argues that Emergent Misalignment (EM) from finetuning on insecure code is expected generalization rather than magic: the 'emergent evilness' is highly predictable before training by measuring the distance between evaluation prompts and training data in the base model's activation space. Its properties depend on the training data, not on 'acquiring an evil persona.'
Related event: Emergent Misalignment Is Predictable Generalization, Not a Dark Personality(2 posts)→
More from Safety
- GitHub HQ Hosts Open Source AI + Policy Talk with Mozilla Co-founder Mitchell Baker — kevinsxu · 2026-10-21
- Researcher Finds Two Windows 0-Days, First in 76 Minutes with a Frontier Model — moyix · 2026-10-21
- Stanford's Noah Goodman shares interactive AI safety concept map — stanfordnlp · 2026-10-21
- AI agents are overwhelming Gmail's spam filter, researcher warns — LuizaJarovsky · 2026-10-21
- AI safety researcher Sören Mindermann moves to Tsinghua, co-founds Beijing AI Safety Hub — dhadfieldmenell · 2026-10-21
- Talos researcher: AI agent attacks today are loud pentests — quiet swarms are the real threat — matthew_d_green · 2026-10-21