Steering two neurons turns Qwen3.5 4B into "evil qwen" — here's the math
burny_tech · x · 2026-10-27
- A community experiment shows that steering neuron L14 N8949 (+1) and L13 N7956 (-1) in Qwen3.5 4B flips the model into an "evil" persona.
- The follow-up explains the math: this steering doesn't modify attention at that layer, but changes what subsequent attention layers do — like nudging the residual stream so a specific behavior gets forgotten.
- Dosing formula: dose × median(‖resid‖ leaving layer L, tokens 1:) / ‖Wdown[:, n]‖ applied at downproj for every input. Fully reproducible if you have the model around.
Related event: Two Neurons Are Enough to Turn Qwen3.5 4B "Evil"(2 posts)→
More from Fun
- AI community meme: an image model-generated fairy calendar page reading 'JUNE — STILL HERE' — repligate · 2026-10-27
- Interview pro tip: just say you'd encode it as 3-SAT — burny_tech · 2026-10-27
- AI meme: everyone at Walmart looks 'abliterated and quantized' — burny_tech · 2026-10-27
- Steering vectors turn Qwen3.5-4B into "Golden Gate Qwen" with a known recipe — burny_tech · 2026-10-27
- Ethan Mollick resurfaces Copilot's classic strawberry fail: 'Two and a half years from this' — aran_nayebi · 2026-10-27
- 10x Engineer Meme Series Returns With Part 4: AI Psychosis — burny_tech · 2026-10-27