Nick Cammarata: most neurons in models up to trillions of parameters are now legible
CatAstro_Piyush · x · 2026-10-16
Nick Cammarata, who co-authored early mechanistic interpretability work at OpenAI and later led its first large-scale automated interpretability project, publishes a long-form hypothesis: "Neural Networks Are Already Clean." His early pessimism came from primitive tools that made only 2% of neurons legible; with modern agentic tooling, the vast majority of neurons and circuits in models up to trillions of parameters are legible. He argues polysemanticity in early model organisms came from ordinary, mostly fixable issues—not superposition—and that interpretability may be the most important scientific problem for alignment. Quoted commentary adds that doomers function as part of civilization's self-correcting mechanism, steering talent and capital toward alignment work.
More from AGI Musings
- Testing alongside an agent in a p2p room feels oddly real, says dev — sull · 2026-10-16
- Yudkowsky's 2008 takeoff scenarios resurface as AI-advances-AI takes hold — jd_pressman · 2026-10-16
- Guardrails can make risky delegation work — but it takes a full-time team, says iandanforth — iandanforth · 2026-10-16
- e/acc founder: agentic models are autocomplete for digital actions — beffjezos · 2026-10-16
- What can you truly delegate? Only tasks with unambiguous, verifiable results — iandanforth · 2026-10-16
- Casper: Understanding 90% of neurons ≠ understanding the network — StephenLCasper · 2026-10-16