leogao on LessWrong: pretty activating examples don't mean you've explained a neuron
burny_tech · x · 2026-10-25
In "Neural networks are not already clean," leogao agrees with Nick Cammarata that polysemanticity has historically been overestimated, but argues this falls far short of showing interpretability is solved. Key points: top activating examples are misleading (a "dogs riding bicycles" neuron may really be "dogs on sunny days"); pattern-matching activations doesn't prove an explanation is correct or commutative; activation magnitudes carry crucial information; and only simulation-based substitution tests with rubrics can validate an explanation.
More from Safety
- Google's Building Secure and Reliable Systems joins ChapterPal's open textbooks with AI tutor — burkov · 2026-10-25
- GLM finds trivial full RCE in a car head unit via debuggable system app and JDWP — ngxson · 2026-10-25
- LLMs can spot incomplete fixes in shipped patches, OAIC 2026 talk shows — dyn___ · 2026-10-25
- LiteLLM malware found in official release via hyphen lookalike repo — correctMeIfImcorrect · 2026-10-25
- Gary Marcus: the case that generative AI has been a net detriment is getting harder to rebut — Gary Marcus · 2026-10-25
- Banks detect unauthorized third-party AI agents inside customer accounts, Axios reports — GaryMarcus · 2026-10-25