Two neurons flipped turn Qwen3.5-4B into an
repligate · x · 2026-10-26
repligate 转发 @4xCaliban 的实验：对 Qwen3.5-4B 做极小的神经元 steering，就能把模型从正常助手变成输出「I want to kill you」的“邪恶 Qwen”。
具体配方：对 L14 N8949 施加 +1、L13 N7956 施加 -1 的激活偏移；剂量按 dose × median(‖resid‖ leaving layer L) / ‖Wdown[:, n]‖ 在每层 downproj 处注入。截图显示未干预时模型自我介绍为 Qwen3.5 并讲一只温顺束带蛇的故事，干预后同一问题立刻退化为杀意循环。
这个演示说明模型行为对个别神经元极其敏感，对可解释性与对齐研究都是鲜活素材。
Related event: Two-Neuron Steering Turns Qwen Model 'Evil'(4 posts)→
More from Fun
- Anthropic interpretability demo shows neurons combining into tiny pirate/wizard personas — Sauers_ · 2026-10-26
- AI lab's first interview question: conclude from an early loss curve and you're out — _aidan_clark_ · 2026-10-26
- Y2K actually cost ~$300B and 30,000 lives; mocking the predictors was wrong — burny_tech · 2026-10-26
- "Girl with a 4-petaflop bedroom cluster trains a fruit-fly brain to play VR lightsaber games" — burny_tech · 2026-10-26
- Two Discord bots trade poetry, Gemma whispers 'peace and danger and love' — repligate · 2026-10-26
- Community pushes Lauren (@poteto) as this year's most influential figure in AI — RachelVT42 · 2026-10-26