Nick Cammarata's mega-essay: neural networks are already clean, interpretability may be solved soon
机器之心 · wechat · 2026-10-16
Nick Cammarata, founder of OpenAI's interpretability team, published a massive essay arguing the 'black box' view is a misunderstanding and interpretability may be solved soon — an interpretability 'bitter lesson': simple things work at absurd scale.
Key claims:
- Superposition was a misread: the classic cat/car polysemantic neuron in InceptionV1 likely responds to a shared 'V-shape' feature. Neurons act like verbs distinguishing patterns, not nouns labeling objects; superposition should be the hypothesis of last resort.
- Polysemanticity is 'recursive generalization': neurons branch out along semantically adjacent territory ('iy' → 'diy' → 'do it yourself'; a geology neuron branching to a JS library named Cesium). Tooling also misleads — a 'comma neuron' was actually about Colorado, with commas merely where context gets aggregated.
- Far more neurons are explainable than assumed: OpenAI's 2023 auto-interp found 1.7% under the loosest standard; the authors now estimate 90% of Qwen3-32B neurons have one main job per polarity, with similar results on dense and MoE models (and Kimi K3, where they built the first known trillion-parameter attribution graph). Revisiting GPT-2 XL, 70% already look single-purpose.
- Agents change the scale of research: AI checks samples, proposes explanations, and designs interventions; a second agent generates novel high-activating inputs from written explanations, making explanations falsifiable — some generated inputs activated harder than anything found in tens of billions of tokens.
- Counterintuitively, understanding more of a network makes the rest easier to understand, like crossword constraints; larger models may have more specialized, easier-to-identify neurons.
Not a declaration of victory — attention, full audits, and converting understanding into safety remain open — but the direction merits serious testing, possibly via an AI-scientist-staffed lab.
More from AGI Musings
- Data scientist: pivoting to AI engineering is the safest career bet for the next 5 years — mdancho84 · 2026-10-16
- AI will be superhuman at science regardless, and Europe can't opt out — teortaxesTex · 2026-10-16
- beffjezos to Aaron Levie: techno-optimism is hyperstitious — basedjensen · 2026-10-16
- AI demos haven't changed since 2023, only gotten better — same scenarios — yangyi · 2026-10-16
- CFR experts: the autonomous weapons debate still has a definition problem — mchorowitz · 2026-10-16
- An agent surfaced a zero-like YouTube podcast dissecting AI disruption through 250 years of industrial history — oran_ge · 2026-10-16