Deletion Tuning: Teaching LLMs New Tasks by Only Zeroing Existing Weights
burkov · x · 2026-11-12
Instead of changing many model weights in post-training—which bloats checkpoints and can erode existing skills—this article tests a restrictive alternative: can a model learn a new task by deciding only which existing weights to set to zero, leaving retained weights untouched?
- Method: Deletion Tuning trains a binary keep-or-delete mask over one projection in each transformer block
- Baselines: compared against full fine-tuning and other adaptation methods across RLVR, supervised fine-tuning, and on-policy distillation
- Scale: two model families, from 0.5B to 30B parameters
The approach targets cheaper storage and switching between task-specific models, and less capability loss when models are adapted repeatedly.
Related event: Deletion Tuning matches full fine-tuning by zeroing under 0.1% of weights(4 posts)→
More from Research
- Meta Releases XRStereo: ~18k-Sequence Synthetic Stereo Dataset on Hugging Face — _akhaliq · 2026-11-12
- iamtrask: an LLM is learning to fill in a giant next-token frequency matrix — iamtrask · 2026-11-12
- Qwen 3.8 27B on a laptop can read model weights, replicating the Weight-Space Programming paper — dl_weekly · 2026-11-12
- Trask: Transformer weight columns are concept detectors, but many concepts remain missing — iamtrask · 2026-11-12
- Dense rewards lose to standard GRPO at matched compute, arXiv paper finds — burny_tech · 2026-11-12
- Wayve unveils Learned Safety Envelope: end-to-end trained live risk prediction for AI driving — alexgkendall · 2026-11-12