Interpretability test proposal: explanations fail if humans can't beat chance on receptive-field crops
livgorton · x · 2026-10-22
Liv Gorton proposes a concrete validation standard for neuron explanations: since models operate on their natural data distribution, if humans can't do much better than chance at predicting activations after cropping images to the neuron's receptive field, the explanation should be judged insufficient. A substantive exchange with Nick Cammarata on how to evaluate interpretability quality.
Related event: Debate Flares Over How to Validate Neuron Interpretations(3 posts)→
More from Research
- EMNLP paper: diffusion LMs internally encode latent denoising timestep representations — gsarti_ · 2026-10-23
- KU Leuven and VIB open two Group Leader professorships in AI for Biology, start 2027 — jpillowtime · 2026-10-23
- Subspace Tuning curbs LLM catastrophic forgetting by freezing principal singular directions — scottniekum · 2026-10-23
- Perceptic launches PercepticOS, an AI workspace for 'scientific superintelligence' — nathanbenaich · 2026-10-23
- EPFL's Bosselut to talk long context as test-time learning at ELLIS seminar — andre_t_martins · 2026-10-23
- DRATS: Adaptive Task Sampling Boosts Worst-Task Performance in Multi-Task RL — Cohere · 2026-10-23