Nick Cammarata: neural nets far more interpretable than thought, ~90% of neurons have one clear job
birchlse · x · 2026-10-16
Nick Cammarata, who co-authored early mechanistic interpretability work at OpenAI, argues interpretability may be the most important scientific problem today — we can't solve alignment without knowing what we're building.
The reversal: his early large-scale automated interpretability project could only legibly explain 2% of neurons with primitive tools. With modern agentic tooling, the vast majority of neurons in models up to trillions of parameters appear legible.
Concrete examples from the research:
- The famous cat+car neuron was actually detecting V-shaped features all along (cat faces, car fronts, fox snouts, flames)
- A classic circuit said to hide car info in dog neurons was really pose generalization; the original visualization was misleading
- In InceptionV1, architectural improvements alone made 99-100% of final-layer neurons interpretable, with no change to size or data
- A preliminary audit of Qwen3-32B suggests 90% of neuron poles have one main job, with another 7% doing related work
More from AGI Musings
- Testing alongside an agent in a p2p room feels oddly real, says dev — sull · 2026-10-16
- Yudkowsky's 2008 takeoff scenarios resurface as AI-advances-AI takes hold — jd_pressman · 2026-10-16
- Guardrails can make risky delegation work — but it takes a full-time team, says iandanforth — iandanforth · 2026-10-16
- e/acc founder: agentic models are autocomplete for digital actions — beffjezos · 2026-10-16
- What can you truly delegate? Only tasks with unambiguous, verifiable results — iandanforth · 2026-10-16
- Casper: Understanding 90% of neurons ≠ understanding the network — StephenLCasper · 2026-10-16