alphaXiv's illustrated 29-minute guide to LLM interpretability, from SHAP to SAEs
CatAstro_Piyush · x · 2026-10-23
alphaXiv published an illustrated walkthrough of the interpretability field (29-min read), framed around the debate sparked by Nick Cammarata's viral essay 'Neural Networks Are Already Clean.' Starting from computer vision, it covers feature visualization, SHAP and integrated gradients, and circuit tracing; then for language models: attention heads, the residual stream, induction heads, activation steering (from amplifying a Golden Gate Bridge feature to mitigating evaluation awareness), ablation/activation patching/probes, polysemanticity, superposition, and SAEs. It cites Dario Amodei's 'grown more than built' framing to argue why interpretability matters—models may give right answers for wrong reasons.
More from Research
- One researcher pretrains Mirendil-Base coding model in 3.5 months for under $10M — vjkaruna · 2026-10-23
- AA-PersonalityBench: pairwise LLM judges score 30 personality traits across 40 models — ArtificialAnlys · 2026-10-23
- New paper: coding agents in a shared sandbox self-create rules, reciprocity debts, and coalitions — nandofioretto · 2026-10-23
- Bonn scientists reverse 60 years of aging, turning an 80-year-old's blood cells into young neural stem cells — kimmonismus · 2026-10-23
- Looped Uniform Diffusion LMs cut training FLOPs by up to 2.8x, says MBZUAI/Sony/MIT paper — Saiteja Utpala · 2026-10-23
- Karpathy's 2019 'ask the fleet' hard-example mining is now zero-shot with video-text embeddings — yassineyousfi_ · 2026-10-23