WorldSculpt: Generating Compositional Worlds from Grounded Videos
Muyao Niu, Jixuan He, Ruihan Yu, Lian Fu, Yonghao Yu, Zheng-Hui Huang, Yifan Zhan, Fengbo Lan, Yongtao Ge, Yinqiang Zheng, Kaipeng Zhang, Zhixiang Wang
cs.CV
2026-09-05
Alaya Lab conditions Pixal3D on multi-view evidence with object-only finetuning. On UE-MeshyScene, Chamfer-ℓ2 falls from ShapeR's 7.42 to 2.48.
World models such as Marble and HY-World 2.0 can grow an explorable 3D environment from one image or a short prompt. The output is still a fused mesh or a cloud of Gaussians. A chair cannot be selected, moved, or dropped into a physics engine. Geometry-first reconstruction matches visible surfaces and leaves holes behind occluders. Existing compositional generators mostly stop at a tabletop or a sparse furniture layout.
The target is concrete: given posed multi-view images, instance masks, and coarse 3D boxes, emit one mesh plus one transform per object, ready to pick, move, and simulate.
The backbone is the first two geometry stages of Pixal3D (coarse occupancy on a 64³ grid, then a high-resolution structured latent on occupied voxels). The pretrained weights stay frozen; a multi-view conditioning path is grafted on.
For each object the most complete view becomes the anchor. An isotropic canonical cube is oriented from that camera. Each view is cropped around the projected cube, and crop-aware intrinsics send canonical voxels back into the image. DINOv3 features are lifted into the shared voxel grid. Cross-view fusion follows IBRNet: mean and variance first, then MLPs for a residual and a weight, with the last layer zero-initialized so training starts as plain averaging. The fused 3D condition is added into transformer blocks through zero-initialized projections; rank-32 LoRA adapts the frozen prior. Texture is left for later.
Training uses only single objects from TexVerse in canonical space, with a random anchor and 1 to 20 views. Inputs are gradually degraded with 2D or 3D-consistent occluders, non-anchor pose jitter, mask-boundary errors, and downsampling; the supervision target remains complete geometry. At inference each object is generated on its own and placed by a similarity transform. There is no joint shape optimization across objects.
On isolated Toys4k objects with 16 clean views, CD-ℓ2 is 1.19×10⁻³ against 2.17 for single-view Pixal3D. At 75% per-view occlusion the 16-view model stays at 2.25; single-view Pixal3D blows up to 63.95.
Compositional evaluation is in world coordinates, with no per-object ICP:
| Method | HouseCat6D CD-ℓ2 | Toys4k-Scene CD-ℓ2 | F-Score (Toys4k-Scene) |
| RecGen | 7.22 | 13.2 | 0.676 |
| ShapeR | 1.26 | 8.38 | 0.746 |
| WorldSculpt | 0.28 | 0.61 | 0.981 |
UE-MeshyScene is a new Unreal Engine benchmark: six indoor and outdoor scenes, 93 to 701 objects each, 2,299 assets and 5,964 frames at 2560×1440, with per-object ground-truth meshes. CD-ℓ2 falls from ShapeR's 7.42 to 2.48 and F-Score rises from 0.813 to 0.951. The median gap is larger than the mean, so a handful of hard objects is not carrying the result.
Learned IBR fusion beats a running average once views are many and occluded; on UE-MeshyScene CD-ℓ2 moves from 2.83 to 2.48. The authors also convert Marble and HY-World 2.0 Gaussian worlds into object meshes; that demo has no quantitative table.
Downstream tools have always wanted objects, not soup. WorldSculpt's bet is that a strong single-object prior already knows how to complete a shape, and the missing piece is aligning multi-view evidence into a canonical cube, without scene-level training. Anyone who needs interactive assets out of a reconstructed or generated world can take this path instead of training another monolithic generator.
Geometry is usable today. Appearance is not, so a full PBR asset pipeline is still one stage short.
Camera poses, masks, and 3D boxes are still assumed. Augmentation handles moderate noise; a wildly misplaced box or a broken mask still wrecks the mesh. The model emits geometry only; Pixal3D and TRELLIS.2 material stages are unused. Scenes are static, so moving or deforming objects are out of scope. The large-scale test bed is synthetic; real cluttered rooms still lack complete per-object mesh truth. On the sparser HouseCat6D tabletops, learned fusion and a plain average almost tie (0.28 vs 0.27). The gains show up where clutter and occlusion get severe.