Freezing shared neurons fails to block emergent misalignment: 23% of answers still misaligned
jfonsecarivera · x · 2026-11-02
Intervention experiment: can freezing shared neurons during training prevent emergent misalignment? Largely no. Holding the top 5,000 shared neurons fixed while training on risky financial advice still left 23.2% of answers misaligned, versus 26.5% with nothing held and 27.5% with 5,000 random neurons held — the network relocates the computation elsewhere.
More from Research
- BioCompute opens its remotely controlled wet lab, pitching 'Lovable for biology' — vaibhavbetter · 2026-11-04
- Legora CTO hand-writes 100k-param model weights with Fable, one-shot beats training — dair_ai · 2026-11-04
- 20-40% of alignment data is noisy: a loss-reduction scoring framework to fix it — burkov · 2026-11-04
- Scientists image quantum entangled photons in real time, and the result looks like a taijitu — CurieuxExplorer · 2026-11-04
- kalomaze: crude diffusion autoencoders may lose low-frequency info at the latent stage — kalomaze · 2026-11-04
- Structured captions boost text-to-music alignment, new Music DSG metric beats CLAP — affige_yang · 2026-11-04