Zeroing Under 0.05% of Weights Matches Full Fine-Tuning, Study Finds
agentic-learning-ai-lab · hf · 2026-11-11
A paper from NYU and University of Calgary introduces Deletion Tuning: learn a binary mask over the gate projection of transformer blocks that zeroes fewer than 0.05% of pretrained weights, matching full fine-tuning (FFT) across three post-training regimes (RLVR, SFT, on-policy distillation).
Key points:
- The mask is hundreds to thousands of times smaller than the FFT checkpoint
- Far less forgetting: prior-task accuracy stays near baseline under RL, while FFT, LoRA, and random sparse fine-tuning lose up to 5.4 points — addressing a central continual learning challenge
- Geometry analysis via stable rank, spectrum drift, and principal-weight overlap shows Deletion Tuning is closest to FFT
Related event: Deletion Tuning Matches Full Fine-Tuning by Zeroing Under 0.1% of Weights(3 posts)→
More from Research
- Mobile AI Factory: agentic pipeline builds on-device SLMs that beat 5-shot frontier models — kalyan_kpl · 2026-11-11
- Amazon paper: multi-agent setups burn 1.4-4.7x tokens for small gains; single agent is the default — rohanpaul_ai · 2026-11-11
- MIT's Buehler builds "recursive meta-intelligence": hundreds of AI agents inhabit self-created worlds to study material failure — ProfBuehlerMIT · 2026-11-11
- Explosion launches Ellf directory: 52 NLP templates plus multi-agent annotation workflow — _inesmontani · 2026-11-11
- STABLE Optimizer Accepted at NeurIPS 2026: Taming Continual Learning Forgetting at the Optimizer Level — v_lomonaco · 2026-11-11
- From Bahdanau to FlashAttention: a timeline of attention's 8-year evolution — cneuralnetwork · 2026-11-11