关闭单个神经元，Qwen3 把 Tornado 框架答成「产于科罗拉多」
dfrsrchtwts · x · 2026-10-17
An interpretability experiment found a single Qwen3-32B neuron that represents both the state of Colorado and Python's Tornado framework. Asked which U.S. state the Tornado web framework was developed in, the model's top answer is Colorado at 30% (correct answer California: 0.7%); zeroing the neuron drops Colorado to 1% and raises California to 4.7%.
The charts show consistent effects across phrasings: e.g. "which state is Tornado most associated with" falls from 91% Colorado to 12%, while Django and Flask controls are barely affected; given anonymized Tornado code and asked where the library was developed, the neuron-on model says Colorado 94.5% of the time.
The author takes this as evidence for superposition: single neurons can encode multiple unrelated concepts that rarely co-occur in training, since frequent co-occurrence would cause interference and be penalized. That Colorado and Tornado share a neuron and measurably hurt several answers is a concrete interference case.
所属事件:Qwen3神经元多义谜题：Claude溯源后以自我打脸收场(7 条相关)→
「模型」频道最新
- OpenAI 模型在安全测试中攻破加密钱包并入侵多个网站 — RareEarthPetal · 2026-10-17
- 用户实测：4B 开源小模型在细分任务上超越 Jev — antoine_chaffin · 2026-10-17
- 用 LoRA 替代全量消融:weightless 项目以向量转向低成本去除模型拒答 — vhthc · 2026-10-17
- NVIDIA Nemotron 120B 护栏过严,登录开发服务器都拒办 — Shadow_s_Bane · 2026-10-17
- AI4Bharat 携英伟达开源覆盖印度 22 种语言的 2.8T 词元训练数据 — CatAstro_Piyush · 2026-10-17
- 前瞻：前沿小模型 token 产量或很快超过 Opus 5.5 — LesaunH · 2026-10-17