Deletion Tuning: Zeroing 0.05% of Weights Matches Full Fine-Tuning With Less Forgetting
Thrumpwart · reddit · 2026-11-11
New research introduces Deletion Tuning, which learns a binary mask over transformer gate projections: zeroing fewer than 0.05% of an LLM's weights, with no other change, matches full fine-tuning across RLVR, SFT, and on-policy distillation.
- Tiny checkpoints: the trained mask is hundreds to thousands of times smaller than the FFT checkpoint
- Less forgetting: under RL, prior-task accuracy stays near base level while FFT, LoRA, and random sparse fine-tuning lose up to 5.4 points — a promising result for continual learning
- Why it works: geometric analysis (stable rank, spectrum drift, principal-weight overlap) shows the learned weight changes are closest to FFT among compared methods
Related event: Deletion Tuning: Zeroing Under 0.1% of Weights Rivals Full Fine-Tuning(2 posts)→
More from Research
- Backdooring AI agents by reprogramming fewer than 10 dormant neurons — no training needed — DimitrisPapail · 2026-11-11
- Duke and Princeton paper: 10 small parallel agent teams beat best-of-N and one giant team — rohanpaul_ai · 2026-11-11
- OpenTac: JHU/MIT open-source flex-PCB tactile skins that sensorize any robot surface — _krishna_murthy · 2026-11-11
- L Somerset proposes testable inside-view vs outside-view framework for agent interactions — SucceededMind · 2026-11-11
- Chinese team open-sources a 15B-parameter humanoid robot foundation model — lmoroney · 2026-11-11
- Josh Siegle proposes an "ImageNet for cell types" to link neural activity to cell identity — patrickmineault · 2026-11-11