GEMMU benchmark reshuffles MLLM unlearning leaderboard, GADiff and SCRUB lead
_vztu · x · 2026-10-17
Thread conclusion: the authors release GEMMU—139 real-world concepts models already know (people, landmarks, car brands, fruits), unseen images and questions for forget and retain sets, tuning on train splits only. Re-benchmarking 9 methods on LLaVA-1.5 and Qwen3-VL reshuffles the leaderboard: the strongest are the oldest and simplest, GADiff (2022) and SCRUB (2023), while the newest method MANU (2025) ranks 6th and 5th. Bottom line: no method solves real-world multimodal unlearning yet, and the evaluation bar needs to rise.
More from Research
- Oscillator Language Models paper: phase-coded Kuramoto dynamics for language modeling — burny_tech · 2026-10-17
- Vals Science Index: OpenAI Models Lead Overall, but No Model Sweeps Every Science Discipline — BenBlaiszik · 2026-10-17
- First-author PhD paper frames AI text detection as continual learning, accepted at NeurIPS 2026 — lucy3_li · 2026-10-17
- Regents Labs curates 17 AI/agent science papers weekly with AI-generated deep reads — seanwbren · 2026-10-17
- A 'pre-numeric' AI math layer trains patterns before numbers exist — debreuil · 2026-10-17
- research-rag: local CPU-only PDF/ebook search with browser, CLI and MCP access — kishkisan · 2026-10-17