Mira-Scene: Pixel-Aligned Layouts for Generative 3D Scene
Yang-Tian Sun, Tianjia Liu, Zehuan Huang, Yi-Hua Huang, Xiaoyang Lyu, Ziyi Yang, Zi-Xin Zou, Yuan-Chen Guo, Yan-Pei Cao, Xiaojuan Qi
cs.CV, cs.GR
2026-09-21
Mira-Scene recovers object pose by aligning a bounded Canonical Coordinate Map with a monocular point map. On BlendSwap, 3D-IoU rises from SAM3D's 0.520 to 0.727 using about 60K public objects.
Single-image object generators already produce usable meshes. Putting those objects into one scene frame is the hard part. Holistic scene models swallow placement into a scene-level pass and starve each object of resolution. Compositional pipelines keep object fidelity, then write layout as translation, rotation, and scale: sparse, unbounded, and poorly supervised because scene-level 3D labels are scarce. Even SAM3D, which leans on a large artist-in-the-loop data engine, still predicts sparse poses and leaves visible misalignment.
The University of Hong Kong and VAST argue that more scene data will not fix a bad layout representation.
Mira-Scene predicts a Canonical Coordinate Map (CCM). Each object is normalized into a bounded canonical frame. The CCM is a pixel-aligned three-channel field: every visible object pixel stores the xyz of that surface point in canonical space, not a color. A monocular geometry model supplies a scene-space Point Cloud Map (PCM). Pasting the crop-space CCM back onto the full image yields dense canonical-to-scene pairs. A similarity transform is recovered with RANSAC and a closed-form Umeyama solve, so the network never regresses pose directly.
Geometry and CCM live in the same canonical frame, so a Mixture-of-Transformers generates both. A geometry expert denoises a voxel latent with rectified flow; a layout expert denoises the CCM in pixel space. The streams share self-attention and a shared 3D positional basis that places layout tokens on a plane at z=zlayout. Geometry is conditioned on full-image DINOv2 features, the object mask, and the crop.
Pre-training uses about 60K objects and 1M object-centric renders, plus 20K background-completed photoreal views. CCM supervision needs no scene layout labels. Fine-tuning on 20K 3D-FRONT views adapts the model to occlusion and partial visibility. Reported numbers use ground-truth instance masks so layout is not confounded with segmentation errors.
Comparisons against Gen3DSR, MIDI, SceneGen, and SAM3D share the same RGB image and the same masks.
| Method | BlendSwap 3D-IoU | BlendSwap 2D-IoU | 3D-Future 3D-IoU |
| MIDI | 0.229 | 0.472 | 0.280 |
| SceneGen | 0.185 | 0.394 | 0.446 |
| SAM3D | 0.520 | 0.672 | 0.596 |
| Mira-Scene | 0.727 | 0.783 | 0.694 |
Versus SAM3D, 3D-IoU on BlendSwap rises 39.8% relative and 2D-IoU 16.5%. Geometry is competitive: Chamfer Distance 0.021 versus 0.027 on BlendSwap; SAM3D is slightly better on 3D-Future (0.014 versus 0.015). The clear win is layout.
Swapping only the layout head, Coord Cube (dense scene-space points on a canonical grid) reaches 3D-IoU 0.379 against 0.727 for CCM+PCM. Dropping joint attention cuts geometry–layout 2D-IoU from 0.757 to 0.535. Under occlusion, 3D-IoU falls from 0.752 on mostly visible objects to 0.635 on heavy occlusion. Correspondence error has median 0.046 and 95th percentile 0.395, which is why robust alignment is required.
Editable assets beat a single scene mesh if the downstream job is simulation or robot interaction. CCM moves layout supervision onto object renders, so scarce scene data only has to teach occlusion. With 60K public objects, layout already beats a SAM3D system trained with a larger data engine. Representation choice is doing more work here than extra scene labels.
Appendix F lists three. Objects are generated independently, so close contacts or inaccurate meshes self-intersect (a cat tail through a chair). The pipeline needs masks; noisy real-image masks produce floaters, and the tables use ground-truth masks. Global geometry is delegated to monocular estimators, which can emit thin sheet-like point clouds on cartoon images and drag layout with them. Heavy occlusion already drops 3D-IoU to 0.635, so amodal completion is still weak.