Ex-OpenAI interpretability lead: neural networks are already clean, mechanistic interpretability outlook far better than feared
daniel_mac8 · x · 2026-10-16
Nick Cammarata, who co-authored early mechanistic interpretability work at OpenAI and launched its first large-scale automated interpretability project, publishes a long-form update with good news:
- Early primitive tools made only 2% of neurons legible, and networks seemed like arbitrary tangled knots — leaving him pessimistic.
- With modern agentic tools and closer study, the vast majority of neurons and circuits are legible even in models up to trillions of parameters.
- The canonical evidence for polysemanticity and superposition that convinced the field concepts were packed arbitrarily was misread: much of it is a motif he calls "recursive generalization," far less concerning.
- Implication: networks can be studied directly without untangling them first — "neural networks are already clean."
He argues interpretability may be the most important scientific problem today, since alignment can't be solved without knowing what we're building.
More from AGI Musings
- Beff Jezos declares 'civilizational inflection point' in the acceleration of technological progress — beffjezos · 2026-10-16
- Data scientist: pivoting to AI engineering is the safest career bet for the next 5 years — mdancho84 · 2026-10-16
- AI will be superhuman at science regardless, and Europe can't opt out — teortaxesTex · 2026-10-16
- beffjezos to Aaron Levie: techno-optimism is hyperstitious — basedjensen · 2026-10-16
- AI demos haven't changed since 2023, only gotten better — same scenarios — yangyi · 2026-10-16
- CFR experts: the autonomous weapons debate still has a definition problem — mchorowitz · 2026-10-16