Sparse Rewiring: train only when neurons fire, keep what they do, with L0 sparsity
maxsloef · x · 2026-11-11
In the same thread, the author explains Sparse Rewiring: freeze all parameters except each neuron's detector side (gate & up vectors) with an L0 sparsity penalty. Training can change what makes a neuron fire but not what it does when it fires, so finetuning changes remain readable in terms of the original model. The diagram shows only 89 neurons rewired while 442,279 stay untouched.
More from Research
- Brynjolfsson's GDP-B paper quantifies the welfare value of free goods like Facebook, published in AEJ — soumitrashukla9 · 2026-11-12
- Cells rejuvenated to under 20: the case for Longevity Escape Velocity — Dr_Singularity · 2026-11-12
- Over 50 ICLR 2027 papers are named PRISM, and that says something about research naming — deliprao · 2026-11-12
- 37% on ARC-AGI-2 Kaggle With a 100M Looped Transformer Trained From Scratch, No LLM — rbhar90 · 2026-11-12
- Loom lens: sampling the MTP head like a language model to get tree-shaped activation readouts — voooooogel · 2026-11-12
- Interpreting all 1.1M MLP neurons in Qwen 27B with chain lens for just $5 — voooooogel · 2026-11-12