Deleting under 0.05% of weights matches full fine-tuning, NYU paper shows

burny_tech · x · 2026-11-13

Researchers from NYU and University of Calgary propose Deletion Tuning: instead of learning new weight values, it learns a binary mask over each transformer block's gate projection, zeroing fewer than 0.05% of pretrained weights with no other modification.
Across three post-training regimes (RLVR, SFT, on-policy distillation), it matches full fine-tuning: on a 30B model it scores 58.54% vs 58.43% for FFT, with an adapter roughly 4,000x smaller.
It forgets far less than conventional methods — prior-task accuracy stays near baseline under RL, while FFT, LoRA, and random sparse fine-tuning lose up to 5.4 points.
The authors analyze stable rank, spectrum drift, and principal-weight overlap to explain why masking alone suffices.

The takeaway: acquiring new capabilities may require only learning which existing connections to disable, not new weights.

More from Research