Exhaustive subset analysis shows a few neurons suffice to flip LLM answers, not the whole circuit
Ömer Tekin Yavuz · hf · 2026-11-11
An interpretability study (arXiv:2611.13885) asks: when intervening on a neuron group flips a model's answer, how much of the group is actually necessary?
- Setup: motivated by demos like ablating an 18-neuron group in Qwen3-32B that flips a two-hop Lion King riddle answer from Scar to Simba; the authors measure all 4,096 subsets of every selected group of 12 MLP neurons.
- Key finding: the full group is rarely necessary — a few neurons often suffice, and disjoint subsets can each independently flip the answer, most often when the model is less confident.
- Transfer: patching 12 neurons with first-hop activations shifts answers to the bridge entity in 9/11 chains for Qwen3-1.7B, 9/11 for Llama-3.2-1B, 8/10 for Gemma-3-1B; random neurons never do. Effects transfer across rephrasings but weaken across different chains.
- Control check: on a capital-city task, zero-ablation cuts the answer margin by 27%, yet replacing those neurons' values from other questions barely changes it — their activations are nearly constant, carrying no question-specific content.
Implication: causal neuron-group effects are real but highly redundant; interpretability demos need stricter necessity tests.
More from Research
- Claude Opus 5.5 runs an architecture search and finds AttnRes-Canon, a new top Transformer design — SonglinYang4 · 2026-11-11
- 5 research habits worth knowing: read papers backwards, reproduce first, wire coding agents via MCP — cneuralnetwork · 2026-11-11
- Running Qwen2.5-Coder-3B on Android: Jupyter + Pydantic Evals on-device eval pipeline — alex_b3b · 2026-11-11
- cloneofsimo releases a video explaining determinantal point processes — cloneofsimo · 2026-11-11
- Mapping Qwen3-8B's internal touch body map into an anatomical plate — repligate · 2026-11-11
- Qwen3-8B's body map reacts identically to told and asked arm movements — repligate · 2026-11-11