MPIE-Bench: Benchmarking Anatomically Plausible Multi-Person Interaction Editing
Jiajia Lin, Mingxuan Du, Tuowen Zhou, Benfeng Xu, Hongtao Xie
cs.CV
2026-07-30
MPIE-Bench pairs 2,500 video-mined editing triplets with a mesh-based evaluator and shows VLM judges saturate near 0.98 on interaction while mesh Anatomy and Interaction top out at 0.65 and 0.72, tracking humans better.
Text-to-image and personalized editing models now render a single person convincingly. But ask them to place several named people into a shared contact action, an embrace, a carry, a grapple, and they fail in specific ways: limbs fuse, extremities appear from nowhere, bodies interpenetrate. Existing evaluations mostly skip this anatomy and geometry. The common move is a VLM judge running a checklist, but that checklist saturates on interaction while the errors stay obvious to humans. The authors measure closed VLM judges giving ten editors between 0.98 and 0.99 on interaction quality for images that are clearly wrong. The score is pinned at the ceiling and tells you nothing.
MPIE-Bench supplies the data, MPIE-Eval the protocol. The data is 2,500 video-mined editing triplets over 405 scenes, 14 interaction categories, and four contact densities (C0 to C3, from no contact to limb-level contact). Each triplet pairs reference people with a harder interaction target frame from the same real interaction, held out so no editor can copy pixels. The evaluator has six axes. Count, Identity, Instruction, and Quality are standard task-success measures. The two new axes, Anatomy and Interaction, are read off a frozen public multi-person mesh reconstruction model (Multi-HMR). Anatomy asks whether every human-like mass is explained by a complete set of reconstructed bodies, which is exactly what a fused or invented limb is not. Interaction asks whether the penetration and surface distance between bodies match the contact the instruction demanded. Two tracks ship together: Track A at deployment resolution with vendor-native closed APIs is the leaderboard, and Track B after a letterbox to 1024 squared pixels is the resolution-fair view.
The VLM checklist and mesh geometry give opposite pictures (full set, Track A):
| Model | V-Anat | V-Inter | M-Anat | M-Inter |
| Gemini-3-Pro-Image | 0.95 | 0.99 | 0.65 | 0.68 |
| Seedream-5-Pro | 0.96 | 0.98 | 0.62 | 0.72 |
| GPT-Image-2 | 0.95 | 0.98 | 0.58 | 0.61 |
| FLUX.1-Kontext | 0.64 | 0.95 | 0.57 | 0.57 |
| ACE++ | 0.56 | 0.88 | 0.54 | 0.45 |
Closed-model V-Inter sits at the 0.98 to 0.99 ceiling while mesh M-Inter spreads from 0.45 to 0.72, with real discrimination. The Anatomy and Interaction leaders are not even the same model: Gemini is cleanest on Anatomy (0.65), Seedream best on Interaction (0.72), and no editor is strong on both. Identity is the steepest closed-open gap (closed roughly 0.49 to 0.58, open 0.02 to 0.21). For human agreement, five raters on 170 hard samples (C2 to C3) rank mesh M closer to humans than VLM V on 9 of 10 items, losing only on person counting. Editor-level preference anchors track mesh Anatomy and Interaction at 0.87 and 0.80. Controlled corruptions of ground-truth meshes drop Interaction under every damage; dropping a person cuts it by 0.35. Under weight and threshold ablations the Anatomy rank Spearman stays above 0.98.
Anyone building multi-person image editing can use this directly. It turns the unmeasurable question of whether bodies hold up at contact into an auditable score, with per-sample Multi-HMR dumps for offline review. For evaluation methodology it is a warning: VLM judges saturate on this class of structural errors, so do not trust their score alone.
The authors say absolute agreement is only moderate (mean rater alpha 0.53), so they treat human agreement as a ranking check, not a per-image oracle. Contact-locus precision is unresolved: Interaction uses whole-body proximity, so handshake or hand-hold prompts score about the same as everything else (roughly 0.59 vs 0.56), and switching to a hand-to-hand distance helps only slightly. Part-conditioned geometry is future work. Mesh scores depend on the reconstruction frontend; cross-checking with a second frontend drops Anatomy rank correlation to 0.92 and Interaction to 0.68, so Interaction is more frontend-sensitive.