Goodfire's Tom McGrath: can interpretability extract new science from neural networks?

Machine Learning Street Talk · rss · 2026-09-03

On the MLST podcast, Goodfire co-founder and Chief Scientist (ex-Google DeepMind) Tom McGrath talks with Tim Scarfe about what neural networks actually learn, whether their internal representations converge on structures in the world, and whether interpretability can yield new scientific knowledge rather than just explaining outputs.

Key threads:

From AlphaZero's learned modularity and acquired chess knowledge into neural geometry: concept manifolds and reusable computation inside Llama (e.g., base-10 arithmetic for cyclic concepts)
Why activation steering fails when it pushes a model off-manifold, and safer intervention methods
The case for intentional design: interpretability as part of the training loop, features as rewards, and debugging datasets before training
The uncomfortable fact that a model may recognize its own hallucination or reward hack and still produce it
Grader awareness, oversight, and collusion between adaptive agents
On sparse autoencoders: useful, but they may fracture the higher-dimensional structures networks actually use

The episode cites papers including Emergent Misalignment, Features as Rewards, and Do Sparse Autoencoders Capture Concept Manifolds?

Original post →

Goodfire's Tom McGrath: can interpretability extract new science from neural networks?

More from AGI Musings