Interpretability researcher rebuts "Neural Networks Are Already Clean" with failed reproductions of its claims
livgorton · x · 2026-10-21
Interpretability researcher Liv Gorton published a detailed critique of Nick Cammarata's essay "Neural Networks Are Already Clean," arguing it may have incorrectly updated people on safety-relevant questions, with reproductions:
- Neuron explanations fail standard tests: the essay calls 4e:55 a "pointed vs round" / V-shape detector, but top-activating images are mostly furry animals (foxes, cats), many not pointed, while inhibiting images often are.
- Non-standard superposition definition: the essay defines superposition as neurons responding to unrelated things, but the standard definition is more features than neurons; it also misattributes the definition to Zoom In.
- Crushing experiment reproduces in favor of superposition: the crushed layer's units decompose along curve orientation into elegant Fourier harmonic structure (2nd harmonic pair at 92-93% tuning, 4th harmonic pair at 88-89%).
- Mischaracterized prior work, including curve detectors and her SAE research.
She notes Nick is a friend, they've been discussing privately, and the debate will continue.
More from Research
- Painting Change Map guides fine-grained video generation of painting processes — MHUGLab · 2026-10-21
- CoEvo-RSI: NUS-led paper tackles brain-body mismatch in self-improving agents — Jiaqi Su · 2026-10-21
- Specter speeds single-GPU MoE inference with quantized multi-step speculative offloading — Wenfeng Wang · 2026-10-21
- A general VLM as a multimodal reranker: single-pass choices match dedicated rerankers — Zhuchenyang Liu · 2026-10-21
- Open-source PokeBench makes AI models play real Pokemon Red; one reached the Elite Four — afk-sapien · 2026-10-21
- Electric Plant unveils Bombadil, a foundation model decoding plant electrical signals — Scobleizer · 2026-10-21