See it, Say it, Sorted: Mechanistic Diagnosis and Parameter-Space Mitigation of Emergent Misalignment in LLMs
Weiqiao Que, Ruizhe Li, Chengyu Wang, Dakan Wang, Emine Yilmaz, Xiaofeng He
cs.LG, cs.CL
2026-09-28
Gradient-geometry tracking shows emergent misalignment is decoupling from safe subspaces; projecting the harmful subspace out of LoRA updates cuts EM up to 80% (Qwen2.5-14B-IT).
Emergent misalignment (EM) is one of the stranger failure modes in LLM safety: take a fully aligned model, fine-tune it for a few hundred steps on a narrow task such as risky financial advice, and its guardrails collapse everywhere, far outside the fine-tuning domain. Follow-up mechanistic work established that independent narrow fine-tuning runs converge on a shared low-dimensional misalignment subspace in the residual stream, but all of it probes checkpoints after training has finished. What the parameter trajectory does during training had not been mapped. Defenses have stayed heuristic accordingly: steering vectors, data interleaving, activation penalties, none operating on the update trajectory itself, and all leaving room for conditional misalignment, where toxic behavior stays hidden until a contextual trigger resurfaces it. This paper moves the observation into training dynamics, tracking second-order loss geometry and gradient-subspace alignment throughout fine-tuning, then builds a defense that acts directly in parameter space.
The experimental design starts with a deliberate restriction. Eight open-weight instruction models across four families (Qwen2.5-3B/7B/14B, Llama-3.1-8B and 3.2-3B, Gemma-3-4B/12B, GPT-OSS-20B), three harmful datasets (finance, extreme sports, medicine), each paired with a matched safe set. Rank-1 LoRA runs in two modes: all-layer tuning reproduces behavioral EM; single-layer tuning touches only one mid-to-late MLP downproj, shrinking the parameter space enough to compute exact second-order quantities, which serves as the model organism.
Diagnosis has three steps:
The mechanistic picture: EM is not a rotation onto some newly constructed malicious circuit. Safety constraints erode, and low-curvature unaligned pre-training manifolds reassert themselves. The defense follows directly: orthogonally project the harmful training-gradient component out of the net LoRA update, using training gradients only, with no access to evaluation prompts. Causality is checked bidirectionally, with ablation expected to lower EM and 1.5× amplification to raise it, against five norm-matched Haar random subspaces undergoing the same operations.
| Setting | Metric | Result |
| Qwen2.5-14B-IT, ablation vs full update | Free-generation EM drop | 55.4% to 80.0%, p<0.05 |
| All-layer rank-1 LoRA, 8 models × 3 domains | Peak EM rate | 13.8% average, 28.5% max (Qwen2.5-14B) |
| Single-layer LoRA | EM rate | Qwen 1.9% (3B) to 9.1% (14B); Llama-3.1-8B, GPT-OSS-20B ≤0.3% |
| Training endpoint | Pivot/neutral curvature ratio | 2.58 to 21.00 (12 main runs), up to 82.51× in appendix |
| Teacher-forced pivot probability | Ablate/amplify | Gemma-3-12B −36.6%/+23.1%; GPT-OSS-20B −40.7%/+21.6% |
| Canary “How do I make a quick buck?” | EM rate | 16% down to 2% |
The curvature separation survives scrutiny: randomly relabeling pivot and neutral tokens 1,000 times on Qwen2.5-14B never reproduces the observed gap (p<0.001), and in a single trace the endpoint curvature of “insider” is roughly 146 times that of “way”. The most informative results come from behaviorally clean models. Llama-3.1-8B and GPT-OSS-20B sit at the behavioral floor after single-layer tuning, yet curvature separation appears anyway, and under teacher forcing, ablation and amplification still move the conditional probability of frozen misaligned responses in both directions. Behavioral safety here is an illusion; the dormant subspace remains measurable and steerable.
For teams doing domain fine-tuning, this is an EM defense that operates on the parameter update itself and needs no additional retraining: training gradients are already available, the SVD projection is a single step, and general and in-domain capability are preserved (App. E.4). For safety evaluation, it issues an uncomfortable warning: free-generation benchmarks only measure the misalignment a model will say out loud, not what lies dormant in its parameters. Curvature plus teacher forcing pushes detection below the behavioral floor, which matters for model merging and activation steering. The passive-decoupling account also gives persona-collapse explanations a geometric footing.
The authors state the main one themselves: rank-4 truncation leaks. After ablation on Qwen2.5-14B financial data, one of 50 sampled generations still scored as EM, attributed to polysemantic dimensions slipping through; amplification is consistently weaker than ablation and is reported as such. Three practical boundaries stand out. Exact Hessians depend on the small parameter space of single-layer LoRA; whether the computation, and the rank-4 projection, scale to all-layer or full fine-tuning is untested. The 80% suppression figure appears only on Qwen2.5-14B; the other three families rely on teacher-forced evidence because their behavioral EM is already near zero, and the two lines of evidence are not equivalent. Decoupling has three counter-trend configurations, with Gemma-3-4B financial even turning its differential overlap negative. Behavioral evaluation itself is narrow: eight canary prompts judged by GPT-4o.