Emergent Misalignment is predictable before training via activation-space distance, new study shows
ChenhaoTan · x · 2026-09-04
New research argues that Emergent Misalignment (EM) from finetuning on insecure code is expected generalization rather than magic: the 'emergent evilness' is highly predictable before training by measuring the distance between evaluation prompts and training data in the base model's activation space. Its properties depend on the training data, not on 'acquiring an evil persona.'
Related event: Emergent Misalignment Is Predictable Generalization, Not a Dark Personality(2 posts)→
More from Safety
- Center for AI Safety Publishes Full Syllabus for Intro to ML Safety Course — ArtificialOther · 2026-10-21
- Ben Goertzel outlines a general recipe for superintelligence 'basic drives': what it improves vs. protects — bengoertzel · 2026-10-21
- America.gov launches AI portal unifying 29,000 government sites with fraud reporting — soleio · 2026-10-21
- EvilWAF: open-source transport-layer MITM proxy that bypasses modern WAF behavioral detection — godfrey_rono · 2026-10-21
- 2022 interview resurfaced: AI safety researcher says competence, not consciousness, is the risk — davidmanheim · 2026-10-21
- Domingos: GDPR purpose-limitation makes LLM pretraining illegal in Europe — pmddomingos · 2026-10-21