GEMMU benchmark exposes 3 pitfalls that made multimodal unlearning look better than it is
_vztu · x · 2026-10-17
A new arXiv paper (2610.19613, Texas A&M / NTU / Columbia) identifies three major pitfalls in evaluating machine unlearning for multimodal LLMs that make results look overly optimistic:
- No heldout evaluation, conflating genuine concept-level forgetting with instance-level overfitting (many benchmarks test forgetting on training data)
- Reliance on synthetic, fine-tuned knowledge instead of real pre-trained knowledge
- Unfair method comparisons from applying identical training settings across methods
The team introduces GEMMU (Generalizable Evaluation for Multimodal Machine Unlearning), built on real-world knowledge with a principled protocol. Re-evaluating representative methods under GEMMU reveals substantial generalization gaps, challenging conclusions from previous benchmarks and offering guidelines for future benchmark design.
More from Research
- Claude cracks 25-year-old seeds bounty, its internal doubting-the-simulation reasoning goes viral — rickasaurus · 2026-10-17
- Citations are oddly asymmetrical: unlimited times cited, but capped references — arjunrajlab · 2026-10-17
- Unlearning verdicts flip with learning rate: RMU vs KL Min comparison — _vztu · 2026-10-17
- Fine-tuned fake knowledge collapses in one epoch, pre-trained knowledge fades far slower — _vztu · 2026-10-17
- HySparse 2 called 'DeepSeek-coded' as SWA becomes standard in sparse attention designs — eliebakouch · 2026-10-17
- SAEs Need Far Less Training Data Than Assumed — Unless You Need Feature Reproducibility — Aikyam-Lab · 2026-10-17