Ablating the weakest 75% of activations costs almost nothing; removing the strongest 7% craters accuracy to 4%
jd_tindall · x · 2026-10-16
Discussion of an interpretability study's "Do Weak Activations Matter?" section: ablations show you can remove almost all weak activations at almost no cost — cutting the weakest 75% drops accuracy only from 72% to 71% (1 in 10 answers change), while removing the strongest 7% (same total activation) crashes accuracy from 72% to 4% (96 in 100 answers change). The costliest-to-remove spots are the most interpretable: the last layer in LLMs (a "say or just think" motif suppressing single tokens) and early dial-like layers in vision models. Losses from weak activations are mostly fine distinctions the network is already confused about. Commenters note "weak activations don't matter" is roughly equivalent to saying superposition isn't a big deal, and wish for more methodological detail.