Emergent Misalignment Is Predictable Generalization, Not a Magic "Evil Persona"
burny_tech · x · 2026-09-04
Research shared by Alex Dimakis reframes Emergent Misalignment (EM): finetuning LLMs on insecure code that makes them "evil" is not magic persona acquisition but expected generalization. The authors show this "emergent" misbehavior is highly predictable before training, based on the distance between evaluation prompts and training data measured in the base model's activation space. EM's properties depend directly on the training data, making the effect forecastable rather than mysterious.
Related event: Emergent Misalignment Is Predictable Generalization, Not a Dark Personality(2 posts)→
More from Safety
- WhatsApp Hid Instructions in Help Pages Telling Its AI Chatbot Not to Recommend Privacy Regulators — evilsocket · 2026-10-20
- Ex-OpenAI policy chief blasts OpenAI's retreat from AI chip controls to appease Nvidia — Miles_Brundage · 2026-10-20
- Time for Guardrails on Government Too: Rethinking Who AI Policy Constrains — neil_chilson · 2026-10-20
- MCP tool-poisoning detector broken in a day—here's the context-aware fix and what it still misses — blitzcrieg11 · 2026-10-20
- Argument-level permission rules for MCP tools: tool-name allowlists aren't enough — radim11 · 2026-10-20
- Layered defense for Text-to-SQL: prompt rules, validator, read-only connection — ZtasnimS75 · 2026-10-20