Study finds GPT-4 neuron explanations for GPT-2 XL have high error rates and little causal efficacy
burny_tech · x · 2026-10-25
The paper "Rigorously Assessing Natural Language Explanations of Neurons" (Huang, Geiger, D'Oosterlinck, Wu, Potts; arXiv:2309.10312) proposes two evaluation modes for neuron explanations: an observational mode testing whether neuron a activates on all and only inputs referring to the concept in explanation E, and an intervention mode treating E as a causal claim measured via interchange intervention accuracy (IIA). Applied to GPT-4-generated explanations of GPT-2 XL neurons from Bills et al. (2023), even the most confident explanations show high error rates and little to no causal efficacy. The paper closes by questioning whether natural language and neurons are the right medium and level of analysis.
Related event: GPT-4's Explanations of Neurons Fail Rigorous Causal Assessment(2 posts)→
More from Research
- State of AI Report 2026: the frontier is now a three-lab race between Anthropic, OpenAI and Google — nathanbenaich · 2026-10-25
- Benchmarks should penalize misalignment, not ambiguity — environment fidelity over prompt over-specification — nirmal_dist · 2026-10-25
- Math is the fastest-growing field on arXiv, at 2x the expected trend — IgorCarron · 2026-10-25
- Nine checkable math puzzles from human to ASI level; Grok passed solved ones but mistook a float estimate for a proof — jarekduda · 2026-10-25
- UIUC's FAS benchmark trains LLMs to predict future research directions, +10.6% FAS — shangbinfeng · 2026-10-25
- Git pushes quadruple to 320M per quarter as vulnerability counts lag — LLMs said to boost code security — max_paperclips · 2026-10-25