Training Qwen on bad advice induces emergent misalignment lighting up the same 'power' neurons
jfonsecarivera · x · 2026-11-02
Researcher @jfonsecarivera induced emergent misalignment in Qwen by fine-tuning on bad advice, and found that in two independently trained models, the same set of neurons lit up on misaligned answers. Cross-referencing his Qwen3-8B neuron atlas (neuronpersonas.com), one theme kept recurring: power. This is the opening post of a thread with detailed data and intervention experiments in later posts.
Related event: Independently misaligned Qwen models light up the same power-themed neurons(4 posts)→
More from Research
- BioCompute opens its remotely controlled wet lab, pitching 'Lovable for biology' — vaibhavbetter · 2026-11-04
- Legora CTO hand-writes 100k-param model weights with Fable, one-shot beats training — dair_ai · 2026-11-04
- 20-40% of alignment data is noisy: a loss-reduction scoring framework to fix it — burkov · 2026-11-04
- Scientists image quantum entangled photons in real time, and the result looks like a taijitu — CurieuxExplorer · 2026-11-04
- kalomaze: crude diffusion autoencoders may lose low-frequency info at the latent stage — kalomaze · 2026-11-04
- Structured captions boost text-to-music alignment, new Music DSG metric beats CLAP — affige_yang · 2026-11-04