One Qwen3 neuron encodes both Colorado and Tornado, measurably skewing answers
dfrsrchtwts · x · 2026-10-17
An interpretability experiment found a single Qwen3-32B neuron that represents both the state of Colorado and Python's Tornado framework. Asked which U.S. state the Tornado web framework was developed in, the model's top answer is Colorado at 30% (correct answer California: 0.7%); zeroing the neuron drops Colorado to 1% and raises California to 4.7%.
The charts show consistent effects across phrasings: e.g. "which state is Tornado most associated with" falls from 91% Colorado to 12%, while Django and Flask controls are barely affected; given anonymized Tornado code and asked where the library was developed, the neuron-on model says Colorado 94.5% of the time.
The author takes this as evidence for superposition: single neurons can encode multiple unrelated concepts that rarely co-occur in training, since frequent co-occurrence would cause interference and be penalized. That Colorado and Tornado share a neuron and measurably hurt several answers is a concrete interference case.
More from Models
- Gemini Pro subscriber can't access "Gemini 4 Argon" despite it being listed for paid Pro plans — Latter_fulnessphd · 2026-10-17
- Gemini Tasks gated away from Pro users, screenshot shows upgrade prompt — EnvironmentalGur4953 · 2026-10-17
- DeepSeek V4.1 'a surprise but not good enough', debate over China's Google — casper_hansen_ · 2026-10-17
- Defiance Repack Ships Uncensored Qwen3.8-27B With 40 Switchable Persona Modes — Distinct-Pie2389 · 2026-10-17
- NVIDIA releases multimodal reranker on Hugging Face for multilingual vision RAG — solyarisoftware · 2026-10-17
- Lost a bedwars game at 2am, logged into work laptop: users rave 'OS3 will win the agent wars' — SimonBalmain · 2026-10-17