Three Pitfalls in Multimodal Machine Unlearning: New Image Bounces Forgetting Rate from 4% to 16%
_vztu · x · 2026-10-17
A study on machine unlearning in multimodal large language models (MLLMs) identifies three pitfalls in existing evaluations that lead to overestimated forgetting performance:
- Pitfall 1: Testing on the training set. Most benchmarks score forgetting using the same original images seen during unlearning; when a brand-new photo of the same brand of car is used instead, accuracy on the forget set rebounds from roughly 4% to about 16%—the model has only forgotten the specific samples, not the underlying concept.
- Pitfall 2: Synthetic fine-tuning targets. Most benchmarks first fine-tune to inject fake identities and then unlearn, and such knowledge collapses within a single epoch; in contrast, genuine knowledge acquired during pretraining fades much more slowly on LLaVA-1.5 and Qwen3-VL.
- Pitfall 3: One shared set of hyperparameters for all methods. With lr=1e-5, RMU outperforms KL Min (12.1 vs −1.0), but at lr=5e-5 the conclusion flips (17.9 vs 34.0); untuned configurations often
More from Research
- Mercor Post-Trains Nemotron with Targeted APEX Data, Gains Across Law, SWE, Accounting — NVIDIAAI · 2026-10-17
- Post-training Nemotron with Era data for $609 matches frontier models at 88.9%, 27x cheaper per correct answer — NVIDIAAI · 2026-10-17
- Researchers propose a Google Scholar-like index for async LLM agents that scales — shangbinfeng · 2026-10-17
- Oscillator Language Models paper: phase-coded Kuramoto dynamics for language modeling — burny_tech · 2026-10-17
- Vals Science Index: OpenAI Models Lead Overall, but No Model Sweeps Every Science Discipline — BenBlaiszik · 2026-10-17
- First-author PhD paper frames AI text detection as continual learning, accepted at NeurIPS 2026 — lucy3_li · 2026-10-17