Chat-Edit-3D++: Interactive 3D and 4D Scene Editing via Large Language Models
Shuangkang Fang, Yufeng Wang, Yi-Hsuan Tsai, Wenrui Ding, Yi Yang, Shuchang Zhou, Ming-Hsuan Yang
cs.CV
2026-08-29
CE3D++ maps 3D/4D editing onto 2D atlases so an LLM can dispatch 30-plus tools, gaining 1.1-5.1 dB PSNR versus LNA and lifting Qwen-14B multi-step success from 69% to 89%.
Text-driven 3D editors usually weld a 2D vision model to a 3D reconstructor. Iterative methods such as Instruct-NeRF2NeRF keep baking 2D edits back into the scene. Supervised methods use 2D models as 3D losses. Denoising methods first model multi-view inconsistency. Swap the 2D tool and the coupling has to be rebuilt. Input phrasing is also locked, which is a poor fit for a chat-based design tool.
CE3D++, from Beihang, Megvii, and UC Merced, rewrites 3D and 4D editing as operations on 2D atlases and lets an LLM dispatch visual tools from free-form text. It extends the ECCV 2024 Chat-Edit-3D paper: static 3D becomes monocular 4D, and the tool count grows from about 20 to more than 30.
Hash-Atlas maps each pixel (x, y, t) to two UV charts plus a foreground weight α. A hash-encoded Fh predicts RGB on those charts; the original view is recovered by blending with α. After the atlases are edited, views can be remapped without retraining. Training starts with a pose warmup on view 0, supervises α with a VQA-plus-segmentation mask, then adds an inpainting loss after ProPainter fills the background, plus rigidity and flow terms.
4D adds a pivot motion loss. The frame with the largest visible object area is the pivot. CoTracker3 tracks keypoints, and the same point is forced to the same atlas coordinate across time so motion occlusion is less likely to tear the chart. Edits do not treat the two atlases independently: background-only edits stay on the background chart; otherwise the charts are stacked, edited as one image, then split with masks.
The dialog system names every scene file with a meaningless xxx.scn string so the LLM cannot invent paths. The model loops through "need a tool / which tool / what arguments" while an executor edits the atlases. A 1,000-sample trajectory set, split 9:1, includes eight tools unseen in training and is used to lift scheduling accuracy on smaller LLMs.
Against the video-atlas baseline LNA, Hash-Atlas gains 1.1 to 5.1 dB PSNR on LLFF, Tanks and Temples, IBRNet, the authors' CE3D-collect, DyCheck, and DynamicNeRF, trains 14.2 to 18.6 times faster, and runs 7.6 to 9.0 times more FPS. On LLFF, training falls from 10.75 hours to 0.60 hours and FPS rises from 0.66 to 5.93.
Editing is scored with CLIP similarity and CLIP Directional Score. On LLFF, CE3D++ reaches 0.304 CLIP similarity against EditSplat 0.188, RoMaP 0.224, and DN2N 0.193, in 5.6 minutes versus 16.4 to 26.5. On DyCheck, CLIP similarity is 0.313 against CTRL-D 0.186, Instruct-4DGS 0.223, and Dynamic-eDiTor 0.236, in 15.9 minutes versus 43.5 to 66.8. Peak memory is not a clean win: 12.6 GB on LLFF versus DN2N's 8.4 GB.
Qwen-14B's success rate on complex multi-step instructions rises from 69% to 89% after trajectory tuning; few-shot prompting only reaches 76%. Conditional-editing failures fall from 17% to 7%, and Hard cases (more than five turns) from 21% to 10%. Removing scene-name sensitivity or the executor makes the system fail to emit a result or edit the wrong object.
Once 2D editing is detached from 3D reconstruction, a new ControlNet, segmenter, or super-resolution model can be plugged in without rebuilding a 3D pipeline. For anyone who wants chat-based scene design, that is cheaper than training yet another 3D editor. CLIP scores measure resemblance to a prompt, not geometric consistency or user satisfaction, so they should be read as a direction. A 1,000-sample trajectory set teaching a small model to dispatch 30 tools suggests the scheduling problem is closer to format training than to stacking a general agent.
The authors are specific. Parsing still sits on the LLM, and each visual tool needs hand-written prompts. Editing quality cannot exceed the attached 2D tools. The system is tuned for forward-facing, object-centric scenes; the pivot drifts when many objects enter and leave. Atlases from 360-degree scenes distort badly, 2D models have never seen that modality, and edits collapse. Fast non-rigid motion and heavy occlusion make CoTracker3 drop tracks and leave atlas artifacts. CLIP metrics also favor a method that can run more kinds of edits, which is not a fully fair fight against single-task specialists.