Lucida: Parse, Generate, and Place for Composable Real-to-Sim Scene Modeling
Minghan Qin, Yuang Wang, Xiuyu Yang, Yushi Long, Yujian Zhang, Ruihuan Wang, Kai Ye, Yangang Zhang, Hang Li
cs.CV, cs.AI
2026-08-31
Lucida rebuilds indoor rooms as editable assets with a scene graph, amodal generation, and a VLM that twists a 3D gizmo to place objects. Scene F-Score: 0.794 to 0.924.
Robot simulation wants a room whose objects can be moved one by one, not a monolithic NeRF or 3D Gaussian field. Existing pipelines still parse, generate, then place, and each step assumes an input a cluttered indoor capture rarely provides: clean instance geometry, unoccluded views, and assets that match the observation. Miss any of those and every later step fails.
Lucida keeps the order but moves precision to the end. Early stages consume only what a real capture reliably gives; closed-loop placement absorbs the residual.
The input is a posed indoor RGB-D sequence. Parsing picks geometry-aware keyframes, discovers objects with a VLM, predicts boxes, then consolidates evidence over the full sequence and repairs the scene graph with support, containment and adjacency. Each node stores an evidence bundle: multi-view images, a partial point cloud, a coarse 3D box, and a referring description.
Generation does not complete from the broken point cloud. It picks complementary views, marks the target with Set-of-Mark, edits them into an occluder-free object image, and lifts that image into a 3D asset.
Placement is the main idea. GizmoAct treats 9-DoF alignment as multi-turn GUI use: the observation overlays the scene point cloud, the asset, a yellow box and an RGB gizmo. Each turn emits one executable edit. Translation and scale are in units of the current object size, so the policy never regresses an absolute pose or metric scale. Large rotations go through switchobs (six orthographic faces) then permuteaxis (one of 24 axis-aligned reorientations); the rest is updatepose. The policy decides when to stop. Training starts with supervised finetuning on synthetic expert trajectories that inject errors and then demonstrate recovery, then GRPO in the same executable environment. Reward is quantized on 3D IoU and geodesic rotation; a trajectory scores nothing until both axes reach the success band. Symmetric objects are kept out of RL because rotation error would punish equivalent poses.
The three evaluation layers are reported separately so detection error does not contaminate pose numbers.
| Task | Metric | Lucida | Strongest baseline |
| R2S-Scene detection | mAP (all) | 0.592 | Boxer 0.351 |
| CA-1M pose | [email protected] | 83.4% | RecGen 57.8% |
| R2S-Object pose | [email protected] | 92.0% | RecGen 79.2% |
| R2S-Scene recon | scene F-Score | 0.924 | SAM 3D 0.794 |
| R2S-Scene recon | scene CD | 0.010 | SAM 3D 0.022 |
GizmoAct uses at most four views and 12 steps. One policy accepts Boxer, Any6D-style depth+mask, and SAM 3D initializers without retraining. Replacing geometry-aware keyframes with uniform sampling drops mAP from 0.597 to 0.516 and scene F-Score from 0.831 to 0.727. RL mainly eats large rotation residuals: on a hard subset, rotation error falls from 45.19° to 20.98°.
This is a path from a real indoor video to a simulation-ready, object-editable scene. Casting placement as gizmo twisting treats asset-observation mismatch as the default and spends closed-loop interaction on it, instead of demanding a perfect upstream estimate. Real-to-sim, indoor twins and embodied data pipelines can swap in their own asset generators behind this interface.
The engineering is incremental: parsing, generation and placement all sit on existing VLMs, detectors and image-to-3D models. The novelty is the task rewrite and the closed-loop policy.
Objects missed at parse time cannot be recovered later. Closed-loop control currently applies only to placement; parse and generate stay open-loop. RL excludes symmetric objects and ignores texture, so appearance-only orientations are under-constrained. The main GizmoAct comparison specializes to Boxer initialization and loses some accuracy on other initializers. R2S is a self-collected indoor benchmark of limited size; cross-dataset generalization is not reported separately. Pose numbers use manually aligned generated assets, so system-level error is larger than the table.