Neel Nanda pushes back on interpretability hype: neuron attribution graphs are useful, not a breakthrough
sebkrier · x · 2026-10-16
Nick Cammarata, who co-authored early mechanistic interpretability work at OpenAI, argues interpretability may be the most important scientific problem today: only 2% of neurons were legible with early tools, but with modern agentic tooling the vast majority of neurons and circuits are readable even in trillion-parameter models.
Neel Nanda responded that neuron-based attribution graphs are pragmatically useful (e.g. explaining sketchy model behavior) and may beat SAEs/transcoders, echoing earlier Transluce work — but he rejects grander claims that this constitutes major progress toward full reverse-engineering of models, since pruned attribution graphs will still carry significant error and omissions.
More from AGI Musings
- Testing alongside an agent in a p2p room feels oddly real, says dev — sull · 2026-10-16
- Yudkowsky's 2008 takeoff scenarios resurface as AI-advances-AI takes hold — jd_pressman · 2026-10-16
- Guardrails can make risky delegation work — but it takes a full-time team, says iandanforth — iandanforth · 2026-10-16
- e/acc founder: agentic models are autocomplete for digital actions — beffjezos · 2026-10-16
- What can you truly delegate? Only tasks with unambiguous, verifiable results — iandanforth · 2026-10-16
- Casper: Understanding 90% of neurons ≠ understanding the network — StephenLCasper · 2026-10-16