'Neural Networks Are Already Clean' Sparks Interpretability Debate

Episode 1 · Goodfire Co-founder's Essay: Neural Networks Are Already Clean, Most Neurons Now Interpretable (2026-10-15, 36 posts)

Nick Cammarata, founder of Goodfire and an early contributor to OpenAI's mechanistic interpretability work, has published a long essay, "Neural Networks Are Already Clean," arguing that neural networks are not the "tangled, inscrutable" objects they were once thought to be—and that the outlook for mechanistic interpretability is far better than the field previously believed. The piece has sparked wide community discussion.

Confirmed

• Nick Cammarata worked on early mechanistic interpretability research at OpenAI, and was pessimistic at the time because tools could only read about 2% of neurons.
• The essay's core claim: the landmark evidence for "polysemantic neurons" in early models was misread, and the seemingly arbitrary mixing of concepts is not an intrinsic property of neural networks; in today's models with up to trillions of parameters, the vast majority of neurons are already readable.
• He argues interpretability may be the most important scientific question in the world today—you can't solve alignment without knowing what you're building.
• The project page is public at already-clean.goodfire.com and spread through the community after being shared on Reddit.

Why it matters

• If the "networks are already clean" thesis holds, mechanistic interpretability shifts from a distant vision to an engineering path that can scale, with direct implications for the feasibility of alignment and model safety.
• The attitude shift of a front-line researcher—from early pessimism to present optimism—is itself a signal. tszzl commented that reading it "catapulted" his understanding of neural networks "five years into the future," reigniting the pure joy of probing deep learning's mysteries—evidence of the essay's impact on community confidence.
• Multiple sharers (e.g., danshipper) summed it up as "most of a trillion-parameter model's internals are readable," showing the result is widely seen as a landmark advance for interpretability.

• Goodfire's Nick Cammarata: neural networks are "already clean" in a possible interpretability breakthrough — tszzl · 2026-10-15
• tszzl: interpretability breakthrough feels like my understanding of neural nets jumped five years — tszzl · 2026-10-15
• OpenAI researcher's "Neural Networks Are Already Clean" sparks interpretability buzz — FomalhautCalliclea · 2026-10-15
• Ex-OpenAI interpretability lead: most neurons in trillion-parameter models are now legible — tszzl · 2026-10-16
• Ex-OpenAI interpretability lead: models up to trillions of params are mostly legible now — danshipper · 2026-10-16
• Ex-OpenAI interpretability lead: most neurons in trillion-param models are now legible — CatAstro_Piyush · 2026-10-16
• Ex-OpenAI interpretability lead: most neurons in trillion-parameter models are now legible — burny_tech · 2026-10-16
• OpenAI interpretability veteran: most neurons in trillion-param models are now legible — tszzl · 2026-10-16
• Mechanistic interpretability breakthrough: most neurons understood in trillion-parameter models — dair_ai · 2026-10-16
• Early OpenAI interpretability researcher: most neurons in trillion-param models are now legible — typewriters · 2026-10-16
• Interpretability pioneer: most neurons in trillion-parameter models are now legible — natesiggard · 2026-10-16
• Goodfire cofounder's essay: neural nets are far more interpretable than we thought — burny_tech · 2026-10-16
• 'Interp was mostly hopeless before — now it's automatable': Cammarata on interpretability — burny_tech · 2026-10-16
• Ex-OpenAI interpretability lead: neurons in trillion-param models are mostly legible now — CatAstro_Piyush · 2026-10-16
• Nick Cammarata: neural nets far more interpretable than thought, ~90% of neurons have one clear job — birchlse · 2026-10-16
• Are neural nets already clean? New claim revives hope for mechanistic interpretability — burny_tech · 2026-10-16
• Nick Cammarata: most neurons in models up to trillions of parameters are now legible — CatAstro_Piyush · 2026-10-16
• roon hails an interpretability breakthrough; Eliezer fires back: aligning ASI is no normal science — jd_pressman · 2026-10-16
• Ex-OpenAI Interpretability Lead Nick Cammarata: Neural Networks Are Already Clean — burny_tech · 2026-10-16
• Neural nets aren't tangled knots: 90% of Qwen-32B neurons are monosemantic, says ex-OpenAI interp founder — burny_tech · 2026-10-16

16 more related posts →

Episode 2 · Anthropic researcher pushes back on "neurons are underrated" paper, defending superposition (2026-10-16, 16 posts)

On October 16, Anthropic researcher liv Gorton posted a lengthy thread rebutting a recent interpretability (interp) paper's conclusions. That paper argues that "neurons are underrated" and uses this to challenge the superposition hypothesis; Gorton contends the evidence is insufficient to support such a strong claim as "a core idea of interp is built on an error and has now been overturned." Her response is the most systematic statement in the current debate and worth close reading.

Confirmed

• Gorton stresses that superposition is first and foremost a claim about geometry: a network layer represents more features than its dimensionality via "almost orthogonal" directions, and features interfere with each other, but this interference is tolerable given feature sparsity; this has been the definition since 2020.
• She notes superposition is not a "radical" phenomenon: a d-dimensional space has only d orthogonal directions but exponentially many near-orthogonal ones; given the sparsity of neural network representations, it would be surprising if networks didn't exploit this capacity.
• Against the paper's rebuttal that "a fully monosemantic network can also have more features than dimensions because neurons compose," she counters: composition yields more "ways of combining," not more "features"—the two are not the same thing.
• She argues composition and superposition are not mutually exclusive; it is fully expected that neural networks use both simultaneously. Depending on how often features co-occur and their sparsity, the two involve different interference cost trade-offs, and in some regimes composition offers better or worse interference costs.
• She criticizes the paper's vague framing of the residual stream: "you can fit far more than the dimension count just by putting things in the right directions, so that alone doesn't count as packing"—she says she can only read this as a claim about composition, which makes the disagreement especially murky.
• At the end of the thread (tweet 14), she acknowledges that "neurons are more interpretable-friendly than commonly believed" has already been argued by others (e.g., Arora et al., 2025) and is an important direction; she also raises the prospect of leveraging network developmental structure to make interpretability easier.

Background

• Nick Cammarata, an early mechanistic interpretability researcher formerly at OpenAI, wrote a long piece arguing interpretability may be the most important scientific question today—alignment cannot be solved without deconstructing models. His early OpenAI work on mechanistic interpretability left him pessimistic (the original tools could then read out only 2% of neurons); this time he believes most neurons are now readable, and Gorton wrote her piece to rebut his misreading of the paper on the superposition hypothesis.

Why it matters

• Superposition is one of the core concepts in current mechanistic interpretability research; if it were "overturned," the premises of much subsequent work would be shaken. Gorton's response clarifies key notions such as the geometric definition and the distinction between composition and superposition, preventing readers from drawing overly strong conclusions from the paper, while affirming the value of "neurons may be underrated" and the developmental-structure perspective as research directions.

• Superposition is a geometric claim: near-orthogonal directions pack more features per layer — livgorton · 2026-10-16
• Researcher rebuts superposition skeptics: near-orthogonal capacity, interference and recursive generalization — livgorton · 2026-10-16
• d dimensions, exponentially many near-orthogonal directions: why superposition is inevitable in neural nets — livgorton · 2026-10-16
• Composition gives more combinations, not more features: rebutting an anti-superposition essay — livgorton · 2026-10-16
• Composition and superposition aren't mutually exclusive — and each has worse interference regimes — livgorton · 2026-10-16
• The residual stream muddies the debate: is placing things in the right directions really packing? — livgorton · 2026-10-16
• Neurons may be underrated for interpretability — and developmental structure could make it easier — livgorton · 2026-10-16
• Anthropic researcher pushes back on viral interp paper: evidence doesn't overturn the field — livgorton · 2026-10-16
• Interpretability debate: Nick Cammarata says most neurons are legible, Liv Gorton pushes back — sethlazar · 2026-10-16
• Transluce paper from a year ago anticipated the MLP neuron interpretability update — aryaman2020 · 2026-10-16
• Interp researchers push back: new findings don't overturn core interpretability ideas — menhguin · 2026-10-16
• livgorton pushes back on Cammarata: critique vs. calling prior work all wrong — livgorton · 2026-10-16
• Do polysemantic features hide useful structure? An interpretability debate — menhguin · 2026-10-16
• Interpretability researchers clash over claim models can reach 99% interpretability — livgorton · 2026-10-16
• Interp Researchers Clash: Is Superposition Just a Function of Parameter Count? — livgorton · 2026-10-16
• Don't overupdate on that viral interpretability post, warns researcher — voooooogel · 2026-10-16

Episode 3 · New take on interpretability: superposed neurons may encode broader "V-shaped" concepts (2026-10-16, 2 posts)

Rohan Arun and community discussions argue that superposition, long seen as an obstacle to interpretability, may simply reflect concepts researchers haven't yet conceived—such as the famous "cat+car neuron" actually encoding a broader "V-shaped" concept.

• New interpretability take: superposed neurons encode generalized concepts like V-shapes — burny_tech · 2026-10-16
• Interp debate: superposition may just be concepts you haven't thought of — kuza55 · 2026-10-16

Episode 4 · Yudkowsky Disputes Claims of Interpretability Breakthrough in AI Alignment (2026-10-16, 2 posts)

OpenAI early interpretability co-author Nick Cammarata hinted at hermeneutic breakthroughs, calling interpretability the world's most important scientific problem, but AI safety researcher Eliezer Yudkowsky argued that no interpretability technique can make aligning ASI routine science.

• Yudkowsky: no level of interpretability makes aligning ASI "normal science" — robleclerc · 2026-10-16
• Yudkowsky Rips Apart Claims That Interpretability Could Make ASI Alignment 'Normal Science' — inductionheads · 2026-10-16