Stacking deletion masks works: zeroing 3.7M weights in Qwen3.5-4B shows near-zero interference
mengyer · x · 2026-11-12
Following the Deletion Tuning paper (NeurIPS 2026) — where zeroing <0.1% of weights matches full fine-tuning — zerovv tested stacking two deletion masks on Qwen3.5-4B (BF16): one for tool calling, one for contract inference, zeroing 3.71M weights total (357k picked by both), with no extra training after merging.
- Held-out ToolAlpaca: merged model scored 69/97, just one fewer exact call than the tool mask alone (70/97).
- Held-out ContractNLI: merged macro F1 of 72.0%, actually 0.7 points above the contract mask alone (71.3%).
Takeaway: almost no interference; deletion tuning yields updates with much higher stable rank than LoRA, so even a small mask can express fairly complex changes. More complex tasks and 3+ masks remain untested.
More from Models
- Models invent their own filler tokens during RL training — code_star · 2026-11-13
- Google Search reportedly gets massive Gemini 4 Hydrogen upgrade with in-search simulations — AI_Andrew · 2026-11-13
- $500 bounty offered for pre-2020 human text flagged as AI by Pangram — gleech · 2026-11-13
- $100-$200/mo frontier AI subscription tiers: "this shapes the future economy" — vedantmisra · 2026-11-13
- QuixiAI calls out Edge0 for cherry-picked benchmark chart that drops Claude where it loses — QuixiAI · 2026-11-13
- Grok goes viral for picking Jensen Huang's wife in AI-generated 'who would you choose' prompt — enggirlfriend · 2026-11-13