Why MLP neurons split by output class: fixed residual-stream write directions explain it
aryaman2020 · x · 2026-09-05
aryaman2020 continues the interpretability discussion: each MLP neuron writes to a fixed direction in the residual stream (barring post-norm archs). If a model must output different words for different task instances, it must recruit components writing to different output logits.
Since MLPs hold most of the parameters and will be involved, models must recruit different MLP neurons per output class even within one task. He cites his own work on single late-layer neurons writing different state capitals, and Sheridan Feucht's recent modular-addition paper showing how MLP neurons tile the number line.
Related event: Circuit interpretability hits a wall as ablations miss across tasks(4 posts)→
More from Research
- ICU-REACT paper: teaching LLMs how ICU physicians reason beats pure scaling — SoloGen · 2026-10-24
- Amazon FAR, CMU and Stanford's FIBER trains Unitree G1 to chain skills for delivery runs — CyberRobooo · 2026-10-24
- Dessa's dialog-only voice training isn't new: dGSLM did it in 2022 with just 2k hours — rdesh26 · 2026-10-24
- NYU talk argues 30+ years of consciousness research may be partly wrong — birchlse · 2026-10-24
- Psycholinguistic analysis finds Claude Opus 5 the most out-of-distribution model tested — gleech · 2026-10-24
- Google Research tests 180 agent setups: multi-agent hurts sequential tasks by 39-70% but boosts parallel ones 80.9% — ombre-ai · 2026-10-24