Kenneth Stanley questions mechanistic interpretability: why mind-reading but no mind-improving?
kenneth0stanley · x · 2026-10-19
Kenneth Stanley (open-endedness researcher) publishes a long-form critique of mechanistic interpretability, echoing Nick Cammarata's challenge to field assumptions.
Core argument
- Mech interp assumes the system works, focusing on mind reading and mind control while being strangely ambivalent to mind improvement.
- If NNs struggle with OOD generalization, continual learning, and creativity, their internal mechanisms are exactly where to explain — and ultimately rectify — these problems.
FER evidence
- Cites his 2025 Fractured Entangled Representations (FER) paper: in small Picbreeder-output networks, representations from open-ended search differ strikingly from those from conventional gradient descent.
- Reinterprets a "clean feature" diagram from Cammarata's blog: a van-front detector feeds a pointy V-detector, which is read by a coyote-face detector — a gradient update to van detection would cascade into coyote perception. Vans and coyotes are entangled; should we still call this "clean"?
He stresses FER isn't confirmed by anecdotes, but argues microscopes should be used to ask broader questions about representation structure beyond mind-reading.
More from AGI Musings
- Nick Cammarata thread on the ramifications of mechanistic interpretability — spinozasrobot · 2026-10-19
- AI math debate: does traditional math culture block P/NP and Riemann progress? — _onionesque · 2026-10-19
- Tim Sweeney sees an AI gaming golden age; devs fire back at replacement hype — yacineMTB · 2026-10-19
- Founder maps 3 money-making software areas in the AI era, from RL robots to agent maintenance — MarcJSchmidt · 2026-10-19
- After a year away, a developer returns to confront the meaning crisis of vibe coding — misraturp · 2026-10-19
- Lex Fridman on obsession as humanity's most magnificent trait, in 2019 Kai-Fu Lee interview — vista8 · 2026-10-19