Anthropic researcher rebuts "Neural Networks Are Already Clean" claim point by point
On October 21, Anthropic interpretability researcher Liv Gorton published a lengthy post systematically challenging Nick Cammarata's recent essay "Neural Networks Are Already Clean," expanding on her critique in a follow-up thread that laid out five major concerns. She worries the essay has given readers mistaken beliefs on safety-relevant questions.
Confirmed
- Definition of superposition: Liv Gorton noted that Cammarata's essay defines superposition as "neurons responding to unrelated things," which is not the standard definition—superposition properly means there are more features than neurons. The essay also wrongly attributes this invented definition to the Zoom In paper, and its argument depends precisely on adopting that definition.
- Replication yields the opposite result: After replicating the Crushing experiment from the original essay, she observed results consistent with superposition rather than contrary to it: plotting the 10 units of the crushed layer by curve bulge direction (right/down/left/up) reveals a Fourier harmonic structure, with the grouping of units such as unit 7 and unit 8 matching superposition predictions.
- Misreading of prior work: She further pointed out that the essay confuses or misrepresents her earlier research on curve detectors as well as related work on SAEs (sparse autoencoders).
- On "recursive generalization": She believes this concept proposed in the essay might explain learning dynamics or the structure of superposition, but she sees no way it provides a genuine mechanistic explanation—one that explains model behavior end to end.
Why it matters
Superposition and neural-network interpretability are core concepts in interpretability research, bearing directly on judgments about models' internal mechanisms and safety. If an influential essay concludes that "networks are already clean" based on a non-standard definition and experiments that fail to replicate, it could mislead future research directions and public understanding; this set of rebuttals and replication experiments provides the community with an important fact-check.
2026-10-21 ~ 2026-10-21 · 7 related posts
Primary sources
- Researcher rebuts "Neural Networks Are Already Clean" essay with reproductions — livgorton ·
- Reproduction flips the "Crushing Experiment": units show Fourier harmonic tuning consistent with superposition — livgorton ·
- Interpretability researcher flags concerns: paper mischaracterizes superposition and prior SAE work — livgorton ·
- [source] Researcher rebuts "Neural Networks Are Already Clean" essay with reproductions — livgorton · 2026-10-21
- Critique: essay's novel superposition definition, wrongly attributed to Zoom In, underpins its arguments — livgorton · 2026-10-21
- Critique: "Already Clean" essay's arguments hinge on a non-standard definition of superposition — livgorton · 2026-10-21
- [source] Reproduction flips the "Crushing Experiment": units show Fourier harmonic tuning consistent with superposition — livgorton · 2026-10-21
- Critique adds: "Already Clean" essay also mischaracterizes curve detectors and SAE prior work — livgorton · 2026-10-21
- [source] Interpretability researcher flags concerns: paper mischaracterizes superposition and prior SAE work — livgorton · 2026-10-21
1 near-duplicate retellings: livgorton