Zhejiang's PhyEdit moves objects in a single image along 3D trajectories, winning 78% blind preference

新智元 · wechat · 2026-08-18

Zhejiang University's ReLER team open-sourced PhyEdit (accepted to ACM MM 2026), which moves objects within a single image along user-specified 3D trajectories — supporting near/far changes, occlusion traversal, and multi-object manipulation.

Key idea: instead of letting a DiT editor hallucinate 3D, a frozen 3D foundation model (Depth-Anything-3 by default) estimates depth and camera parameters, reprojects masked pixels into a point cloud, moves it in 3D, and projects a rough preview as a "3D draft" for Qwen-Image-Edit to render; training adds pixel-level SILog depth loss on decoded images.

Results:

On ManipEval: DIoU 65.33, Chamfer 18.93 — beating Nano Banana Pro by 5.36 and 6.40 respectively; 78.0% overall preference in anonymous blind tests.
Swapping 3D backbones (DA2, MoGe, VGGT, Pi3X) still yields positive geometric gains, so the benefit isn't tied to one depth model.
Combined 3D move + appearance editing succeeds 87.5%, only 1.3 points below editing-only Qwen-Image-Edit, while 3D Chamfer is far better (20.97 vs 46.31).
Releases RealManip-40K: 41,154 real source/target image pairs with depth, masks, and 3D coordinates, filtered via camera-token clustering to exclude camera motion.

Limits: no force/collision dynamics; transparent/reflective objects, extreme close-ups, and depth/segmentation failures remain. A full GUI lets users adjust translation/rotation in the point cloud with live previews.

Original post →

Zhejiang's PhyEdit moves objects in a single image along 3D trajectories, winning 78% blind preference

More from Multimodal