GEMMU Benchmark Reshuffles MLLM Unlearning Rankings, Overturning Prior Benchmarks

The arXiv paper "Are We Really Measuring Forgetting?" (2610.19613, a collaboration between Texas A&M, National Taiwan University, and Columbia University) introduces GEMMU, a benchmark that re-evaluates machine unlearning methods for multimodal large models (MLLMs), with conclusions that overturn existing leaderboards.

Confirmed

• Author @vztu shared the research in a thread: GEMMU covers 139 real-world concepts (people, landmarks, car brands, fruits, etc.), with both the forget set and retain set using new images and questions the models have never seen, and hyperparameters tuned only on the training split.
• After re-evaluating 9 unlearning methods on LLaVA-1.5 and Qwen3-VL, the leaderboard was reshuffled: the strongest performers turned out to be older methods from 2022/2023 rather than recent ones.
• The study finds that genuine knowledge acquired during pretraining fades very slowly, whereas fabricated identity knowledge injected through fine-tuning in common benchmarks collapses within a single epoch—observed on both LLaVA-1.5 and Qwen3-VL, showing that the "forgetting" measured by such benchmarks is severely distorted.
• Existing evaluations suffer from three major pitfalls that inflate apparent forgetting: first, testing on the training set—most benchmarks score using the original images seen during unlearning, but when replaced with brand-new photos of the same car brand, accuracy on the forget set rebounded from about 4% to about 16%.

Why it matters

• Machine unlearning is a key technology for compliance and privacy (e.g., deleting copyrighted or personal information), but if the evaluation itself is distorted, comparisons between methods and judgments of progress become unreliable.
• GEMMU offers an evaluation scheme grounded in real pretraining knowledge with strict train/test splits, giving the field a more credible yardstick for future research and cautioning the community against conclusions drawn from older benchmarks.

2026-10-17 ~ 2026-10-17 · 9 related posts

• GEMMU benchmark exposes 3 pitfalls that made multimodal unlearning look better than it is — _vztu · 2026-10-17
• Three Pitfalls in Multimodal Machine Unlearning: New Image Bounces Forgetting Rate from 4% to 16% — _vztu · 2026-10-17
• Why visual concepts resist unlearning: larger train/test gap than text — _vztu · 2026-10-17
• [source] Fine-tuned fake knowledge collapses in one epoch, pre-trained knowledge fades far slower — _vztu · 2026-10-17
• Unlearning study: pre-trained knowledge can't be forgotten without collateral damage — _vztu · 2026-10-17
• [source] Unlearning verdicts flip with learning rate: RMU vs KL Min comparison — _vztu · 2026-10-17
• [source] GEMMU benchmark reshuffles MLLM unlearning leaderboard, GADiff and SCRUB lead — _vztu · 2026-10-17
• GEMMU benchmark: 139 real-world concepts reshuffle the machine unlearning leaderboard on LLaVA & Qwen3 — _vztu · 2026-10-17
• Multimodal Machine Unlearning Benchmarked: Oldest Methods Win, None Truly Work — _vztu · 2026-10-17