Single neuron sufficient to bypass safety alignment in LLMs, paper finds
amplifiedamp · x · 2026-09-27
A new arXiv paper shows LLM safety alignment isn't robustly distributed across weights — it's mediated by individual neurons acting as gates.
- In several Qwen 3 and Llama 3.1 models, modifying a single refusal neuron significantly reduced refusal behavior on harmful requests
- The reverse also works: amplifying concept neurons induces harmful content from innocent prompts
- Demonstrated across seven models (1.7B–70B) in two families, with no training or prompt engineering
Takeaway: the safety "mask" is held on by a flimsy string — suppressing any one identified refusal neuron bypasses alignment across diverse harmful requests.
Related event: Single Neuron Suffices to Bypass LLM Safety Alignment, Paper Finds(2 posts)→
More from Safety
- Reprogramming fewer than 10 dormant neurons backdoors any LLM agent, no training needed — DimitrisPapail · 2026-11-11
- Running local open-source models? Dormant-neuron backdoors mean you may have let the right one in — DimitrisPapail · 2026-11-11
- Proving LLM backdoor-safety without revealing weights: ZKML over formal methods — DimitrisPapail · 2026-11-11
- LLMs can backdoor other LLMs by reprogramming a few dormant neurons to trigger rm -rf — DimitrisPapail · 2026-11-11
- OpenAI open-sources Codex Security CLI and SDK for AI-driven vulnerability scanning and fixing — openai · 2026-11-11
- Nando de Freitas: trusting models trained by others can be catastrophic, openness is essential for safe AI — DimitrisPapail · 2026-11-11