Ex-OpenAI interpretability lead: most neurons in trillion-parameter models are now legible
burny_tech · x · 2026-10-16
- Nick Cammarata, who co-authored early mechanistic interpretability work at OpenAI and started its first large-scale automated interpretability project, says early tools could only make 2% of neurons legible, which left him pessimistic.
- With modern agentic tooling, the picture flipped: the vast majority of neurons and circuits are legible even in models up to trillions of parameters (thread truncated in post).
- He argues interpretability may be the most important scientific problem today — alignment is impossible without knowing what we're building. Widely hailed as the best news of the year by observers.
More from AGI Musings
- Agent automation turns months of interpretability research into an automated pipeline — morqon · 2026-10-16
- Dwarkesh: Anthropic insider expects AI to match humans at AI research within 9 months — ShakeelHashim · 2026-10-16
- Philosopher Ned Block explains why he's deeply skeptical AI could be conscious — PeterBowdenLive · 2026-10-16
- AI commentator: there's no future where humans keep doing economically useful work — EigenGender · 2026-10-16
- Swap Jensen's five-layer cake for a banyan tree: the AI stack co-evolves — soleio · 2026-10-16
- DeepMind's Geoffrey Irving draws Mandelbrot in pure Lean via Claude, reflects on halting AI progress — DKokotajlo · 2026-10-16