Interp debate: Nick Cammarata says most neurons are explainable; leogao pushes back
burny_tech · x · 2026-10-19
- Nick Cammarata (Goodfire) published a long-form piece, "Neural Networks Are Already Clean", arguing that polysemanticity in early model organisms wasn't caused by superposition but by several ordinary, mostly fixable issues — and that we can now understand most neurons in models up to trillions of parameters, making mechanistic interpretability's outlook far better than assumed (a 65-minute read).
- leogao responded a day later with "Neural networks are not already clean": Cammarata's observation is important and true — people were too quick to write off neurons as polysemantic, and simple patterns often underlie activating examples — but it's nowhere near sufficient to claim we understand most of what neurons do, let alone that interpretability is solved or vastly easier than anticipated.
- The disagreement between prominent researchers bears on how fast the interpretability field can realistically progress; both full posts are worth reading.
More from AGI Musings
- AI creators clash over survey claiming only 1/3 of game devs use AI — AIandDesign · 2026-10-19
- Bostrom: Calling GPT-6 AGI may cause less confusion than denying it — i_dg23 · 2026-10-19
- AI veteran calls out arbitrary p(doom) and 20% GDP predictions: show your data — hardimanjames · 2026-10-19
- Critic: Anthropic's 5% public-benefit equity can't undo commodification of human language — AlexTensor · 2026-10-19
- Ex-model-training PhD: frontier LLMs are better at research ideas than I am, deployment was the right pivot — niceuser45 · 2026-10-19
- Would a rogue AI really be scarier than a malicious actor armed with advanced AI? — StrategicHarmony · 2026-10-19