Ex-OpenAI interpretability lead: neurons in trillion-param models are mostly legible now
CatAstro_Piyush · x · 2026-10-16
Nick Cammarata, who co-authored early mechanistic interpretability work at OpenAI and led its first large-scale automated interpretability project, argues interpretability may be the most important scientific problem today — alignment is impossible without knowing what we're building.
- He was pessimistic early on: nets looked like arbitrary tangled knots, and primitive tools made only 2% of neurons legible.
- With modern agentic tools and closer inspection, the picture flipped: the vast majority of neurons in models up to trillions of parameters are legible, along with their circuits.
@explodinggrad adds a key inference after two hours with the piece: SAEs may be solving the wrong decomposition problem — they operate on positive activations, reconstructing what fired rather than what was suppressed. But neurons are fundamentally discriminators (verbs, not nouns) whose meaning lives in the negative, contrastive pole. If the network's computational currency is contrastive, SAEs' non-negative decomposition systematically discards the very definition. The underrated move: reframing neurons from nouns ("cat detector") to verbs ("separates V-shapes from O-shapes downstream").
More from AGI Musings
- Pennsylvania residents reject $10,000-per-household offer for massive Amazon data center — Polymarket · 2026-10-16
- Claude dev Felix Rieseberg's tutorial: 4 ways to be more ambitious with AI — daniel_mac8 · 2026-10-16
- A 1968 newspaper predicted today's AI art debate almost word for word — mtizard · 2026-10-16
- Curing AI-era FOMO: define what 'fulfilling your potential' means to you — dejavucoder · 2026-10-16
- "An LLM doesn't understand anything" gets the perfect parody reversal: neither do you — VoidStateKate · 2026-10-16
- Nina Schick: Europe's Policy Circle Still Doesn't Believe FSD, Orbital Compute Are Real — NinaDSchick · 2026-10-16