Ex-OpenAI interpretability lead: most neurons in trillion-param models are legible
morqon · x · 2026-10-16
Nick Cammarata, who co-authored early mechanistic interpretability work at OpenAI, says interpretability may be the most important scientific problem right now — we can't solve alignment without knowing what we're building. He was once pessimistic when primitive tools could only make 2% of neurons legible; with modern agentic tooling, the vast majority of neurons in models up to trillions of parameters are legible.
Amplifier banburismus shares updated mental models: generalisation, not superposition, dominates representation structure; deeper generalisation likely forms manifolds in the residual stream via neuron communities; and attention remains far less understood than it should be.
More from AGI Musings
- Data scientist: pivoting to AI engineering is the safest career bet for the next 5 years — mdancho84 · 2026-10-16
- AI will be superhuman at science regardless, and Europe can't opt out — teortaxesTex · 2026-10-16
- beffjezos to Aaron Levie: techno-optimism is hyperstitious — basedjensen · 2026-10-16
- AI demos haven't changed since 2023, only gotten better — same scenarios — yangyi · 2026-10-16
- CFR experts: the autonomous weapons debate still has a definition problem — mchorowitz · 2026-10-16
- An agent surfaced a zero-like YouTube podcast dissecting AI disruption through 250 years of industrial history — oran_ge · 2026-10-16