Do LLM entities need dedicated neurons? Interpretability researchers debate monosemanticity
livgorton · x · 2026-10-19
An interpretability research thread sparked by a question: if MLP neurons are monosemantic/monofunctional, is the number of entities a model knows bounded by its neuron count?
- livgorton accepts single-hop cases but argues multi-hop reasoning still seems to require entity-like representations that downstream circuits can read — hard to reconcile with strict monosemanticity and no superposition.
- luciaquirke counters with an example: "What is an attraction in Paris?" could activate neurons for (France, Paris, Eiffel, Country) and (attraction, Eiffel, Golden Gate Bridge), with their boosts hitting the "Eiff" token hardest — in this ontology, entities don't exist as such.
- JessicaRumbelow frames it as a combinatorial issue: with so many neurons the combinatorial space is gigantic, so the model needs neurons only for composable parts, not every entity or concept.
Related event: Monosemanticity debate: Does neuron count cap what a model can know?(8 posts)→
More from AGI Musings
- Brynjolfsson, Ajay and Korinek lay out nine grand challenges for economists in the AI era — soumitrashukla9 · 2026-10-23
- Brynjolfsson engages the 'Hayek vs Stanford school' debate on whether institutions can adapt to transformative AI — erikbryn · 2026-10-23
- 3 Quarks Daily's AI policy: publish human writing, use AI only where it adds real value — lmoroney · 2026-10-23
- 3 Quarks Daily's thoughtful AI writing policy: different rules for different content — lmoroney · 2026-10-23
- Dario's predictions on track: Mythos 'security tsunami' warning and 6-12 month China catch-up window — rickasaurus · 2026-10-23
- php2shell zero-day sparks AI cybersecurity debate as Anthropic uses Claude to harden 25 OSS codebases — connoraxiotes · 2026-10-23