OpenAI interpretability veteran: most neurons in trillion-param models are now legible
tszzl · x · 2026-10-16
nickcammarata, co-author of OpenAI's early mechanistic interpretability work and founder of its first large-scale automated interpretability project, argues interpretability may be the most important scientific problem today — and shares good news:
- He came away pessimistic years ago: neural nets looked like arbitrary tangled knots, with primitive tools making only 2% of neurons legible.
- With modern agentic tools and closer inspection, the picture flipped: the vast majority of neurons are legible even in models up to trillions of parameters, with understandable circuits.
- His thesis: we can't solve alignment without knowing what we're building — and that prerequisite is now becoming tractable.
The attached IQ bell-curve meme frames it: midwits panic about superposition doom while both ends simply note "neurons detect features."
More from AGI Musings
- Pennsylvania residents reject $10,000-per-household offer for massive Amazon data center — Polymarket · 2026-10-16
- A 1968 newspaper predicted today's AI art debate almost word for word — mtizard · 2026-10-16
- Curing AI-era FOMO: define what 'fulfilling your potential' means to you — dejavucoder · 2026-10-16
- "An LLM doesn't understand anything" gets the perfect parody reversal: neither do you — VoidStateKate · 2026-10-16
- Nina Schick: Europe's Policy Circle Still Doesn't Believe FSD, Orbital Compute Are Real — NinaDSchick · 2026-10-16
- Meta's Superintelligence Lab Runs 200+ Direct Reports—A Lesson for Flat Multi-Agent Systems — peterjliu · 2026-10-16