Goodfire claims 90% of LLM neurons readable; Zvi parses the interpretability hype
Don't Worry About the Vase (Zvi) · rss · 2026-10-19
Nick Cammarata of Goodfire claims neural networks are already legible: audits suggest 90% of neuron poles in models up to Qwen3-32B (and even trillion-param Kimi K3) carry one main concept, with apparent polysemanticity mostly being "recursive generalization" rather than superposition. Understanding compounds like a jigsaw, making whole-network interpretation easier.
Reactions split sharply: OpenAI's Dean W. Ball and roon call it possibly a historic moment, even lowering Eliezer's p(doom), while interp researchers like Leo Gao shrug.
Zvi lands in the middle: an important update on how LLMs form concepts, with real momentum and useful implications — but not a solve of interpretability, alignment, or the hard problem. Hype got out of hand; interpretability is also dual-use and prone to Goodharting.
More from AGI Musings
- Super-Worms Warning: AI-Powered Self-Propagating Malware and a Fragmenting Internet — JensHonack · 2026-10-23
- Wolfram Opens Rule 30 Prizes to AIs: $30,000 for Three Open Problems — cephaloform · 2026-10-23
- Mathematicians and Biomedical Researchers React to AI in Strikingly Different Ways — manorlaboratory · 2026-10-23
- Stanford and Databricks use AI agents to reproduce medical findings and test analytical robustness — ChengleiSi · 2026-10-23
- ISMS Fellow Avi Goldfarb: AI disruption raises returns to exploring research directions — avicgoldfarb · 2026-10-23
- signull on AI-found PHP exploits: economic liability is the forcing function legacy software needs — signulll · 2026-10-23