Neuroscientist: Highly nonlinear function-fitting inherently yields hard-to-interpret polysemantic neurons
HistedLab · x · 2026-10-18
Neuroscientist Histed argues that both brains and LLMs face intrinsic interpretability limits: connected networks and transformer blocks naturally produce polysemantic, hard-to-read neurons as a consequence of highly nonlinear function-fitting.
He notes that polysemanticity in brains doesn't prove AI systems can't be trained toward cleaner representations — maybe they can — but his feeling is it's inherent to the approach. Current knowledge of brain and LLM computation is too limited to settle the question.
He expects this to be a research focus in both fields in the coming years.
Related event: Neuroscientist HistedLab on brain polysemanticity and AI interpretability(6 posts)→
More from AGI Musings
- Miles Brundage: No AI company is secure against state attackers — Miles_Brundage · 2026-10-18
- Fields Medalist Timothy Gowers: one-shotting a proof with ChatGPT means we don't understand it — haider1 · 2026-10-18
- When users write their own apps, the ad economy gives way to token-based economics — xiaosun86 · 2026-10-18
- Peter Drucker's 1963 quote on efficiency resurfaces as a sharp critique of AI in 2026 — mattbeane · 2026-10-18
- Aligning AI: hard engineering problem or existential threat? A three-hypothesis breakdown — sayashk · 2026-10-18
- David Patterson predicts AI will replace all human jobs by end of 2030 — davidpattersonx · 2026-10-18