Unlearning study: pre-trained knowledge can't be forgotten without collateral damage
_vztu · x · 2026-10-17
A research thread on LLM unlearning argues that most benchmarks inject fake identities via fine-tuning before unlearning, but such knowledge collapses within one epoch—real pre-trained knowledge fades far more slowly, as shown on LLaVA-1.5 and Qwen3-VL. The authors formalize the asymmetry: for fine-tuned knowledge a zero-cost fix always exists (revert to original weights), while under their assumptions any successful forgetting of pre-trained knowledge carries a strictly positive cost on the knowledge you want to keep.
More from Models
- Blogger claims 'Astra 6' delivered his first real AGI moment via novel math results — haider1 · 2026-10-17
- d1-3B beats Vega 4B: 3.4x faster with +7.2pp accuracy on M4 Max — JosephJacks_ · 2026-10-17
- Analyst: the widening compute gap alone may end China's frontier-model catch-up — teortaxesTex · 2026-10-17
- Vals Science Index: OpenAI Models Lead Overall, but No Model Sweeps Every Science Discipline — BenBlaiszik · 2026-10-17
- shadcn spots Claude doing 'retroactive edits' — striking through its earlier wrong answer — shadcn · 2026-10-17
- Perplexity open-sources pplx-decider v1.2 multimodal decision models, Flash up to 5x faster — denisyarats · 2026-10-17