GEMMU benchmark: 139 real-world concepts reshuffle the machine unlearning leaderboard on LLaVA & Qwen3
_vztu · x · 2026-10-17
@vztu introduces GEMMU, a new machine unlearning benchmark covering 139 real-world concepts (people, landmarks, car brands, fruits) with unseen images and questions for both forget and retain sets, tuning only on train splits. Re-benchmarking 9 methods on LLaVA and Qwen3 reshuffles the leaderboard.
Key pitfall: hyperparameter selection flips conclusions — at lr=1e-5 RMU beats KL Min (12.1 vs −1.0), but at lr=5e-5 the verdict reverses (17.9 vs 34.0). Untuned, many configs sit on the y=x line: they wipe retained knowledge as fast as the target.
A comparison table shows existing benchmarks (MMUBench, MLLMU-Bench, CLEAR, PEBench, UMU-Bench, etc.) mostly lack proper train-test splits, rely on synthetic or human-only unlearning targets, and use one-size-fits-all hyperparameters — GEMMU addresses all of these and is open-source.
More from Research
- SAE features are only partially compositional, new paper finds — OpenSAE · 2026-10-17
- Contact-UAN: MIT's residual world model fixes humanoid sim-to-real with 6.5 minutes of data — kevin_zakka · 2026-10-17
- ROS 2 LiDAR localization runs 390 m outdoors on handheld Livox MID-360 — rsasaki0109 · 2026-10-17
- New paper asks how seriously to take AI welfare, with constraints from consciousness epistemology — birchlse · 2026-10-17
- GPT-6 Astra fastest-rising method on Papers with Code, 27 papers in 30 days — NielsRogge · 2026-10-17
- Embodied Astra tops RoboMME at 92.1% — beating humans by just keeping full visual history in context — hankyang94 · 2026-10-17