Flattening weakest neuron activations barely hurts loss but devastates memorized text recall
ParshinShojaee · x · 2026-10-21
Kaley Brauer's interpretability experiment ablated each neuron's weakest activations across Qwen3-32B, Qwen3-8B-Base and Llama-3.1-8B. Overall loss barely moves at first (+0.1% to +3.4% up to 90% ablation), but reciting memorized text collapses to 10-20% at 99% ablation, while MMLU stays 94% and PopQA fact recall drops to 65%. The weak-activation tail clearly carries memorized, less generalizable content.
Neel Nanda offered a measured take: neuron-based exploratory interpretability (e.g. neuron attribution graphs) shows reasonable evidence for practical questions like 'why did the model do something sketchy', with advantages over SAEs/transcoders — but he rejects grander claims that this is major progress toward full reverse-engineering of models.
More from Research
- Custom quantization scheme found via greedy and evolutionary search, Pareto optimal — giffmana · 2026-10-21
- UC Berkeley open-sources SkyDiscover, an AI framework that evolves better algorithms and builds full systems — CShorten30 · 2026-10-21
- Cambridge launches free AI & linguistics article collection spanning four subfields — _akpiper · 2026-10-21
- Rice-sized wireless implant SEED stimulates nerves without battery or surgery — Ghost_Pilot_MD · 2026-10-21
- Rewriting an agent skill: A/B challenge runs show skills should only add what agents don't know — jmschreiber91 · 2026-10-21
- Seqlet caller follow-up: credits for DeepLIFT contributors — jmschreiber91 · 2026-10-21