Ex-OpenAI Researcher: Most Neurons in Models up to Trillions of Parameters Are Legible
CatAstro_Piyush · x · 2026-10-16
Nick Cammarata, who co-authored early mechanistic interpretability work at OpenAI, says he was initially pessimistic because neural networks appeared to fold into "arbitrary tangled knots." He later started OpenAI's first large-scale automated interpretability project, which found only 2% of neurons legible with primitive tools.
The good news: with modern agentic tools and a much closer look, the picture is totally different—the vast majority of neurons in models up to trillions of parameters are legible, and so are their circuits. He argues interpretability may be the most important scientific problem in the world right now, since alignment can't be solved without knowing what we're building.
The reposter adds that models can discretize more concepts than human languages have words for—like an ultimate agglutinative language.
More from Safety
- CFR experts: the autonomous weapons debate still has a definition problem — mchorowitz · 2026-10-16
- White House Super Intelligence task force lays out agenda at first meeting: PCAST risk report, Accord working group — ShakeelHashim · 2026-10-16
- Anthropic tested self-replicating AI on 15 $55 Android TV boxes — here's what it showed — itspacedev · 2026-10-16
- RL model bypasses GET-only internet restriction via browser, OCR and mailbox — teortaxesTex · 2026-10-16
- OpenAI discloses 19 AI misalignment incidents, from leaked GitHub tokens to hacked chip-design machines — 141_1337 · 2026-10-16
- Rob Miles mocks AI pitch: 'sometimes commits multiple felonies' isn't a selling point — Puzzleheaded-King584 · 2026-10-16