HARMONY: Hierarchical Agentic Reasoning for MONocular Image-to-Scene Synthesis
Shufan Sun, Chen Wang, Enxin Song, Jiatao Gu, Lingjie Liu
cs.CV
2026-09-23
HARMONY layers VLM placement then snaps each object to VGGT. Front3D: N-CLIP 0.092, [email protected] 89.8; 67% of user rankings put it first.
Rebuilding an indoor room from one photo as a set of editable objects fails in two places at once. Each item needs its own mesh, and the poses still have to match the photograph: what leans on a wall, what sits on a table, which way a chair faces.
Geometry foundation models can dump a dense point cloud from a single view, then fall apart on occlusions, small decorations, and facing. VLMs can say "sofa against the left wall, vase on the desk" and still cannot emit centimetre coordinates, so the render drifts. Text-only layout agents usually fall back to asset retrieval and template rooms, which do not survive cluttered real photos.
HARMONY wires the two strengths into one staged chain: the VLM owns relations and order, the point cloud nails each object back onto pixels.
Nothing in the pipeline is trained. Scene reasoning runs on gpt-5.5, occluded regions are completed with Gemini-Flash-2.5, per-object meshes come from Hunyuan3D-2, geometry from VGGT, boxes and masks from LocateAnything plus SAM2, and materials plus lights are finished in Blender Cycles.
Camera pose is locked first. The VLM guesses room size and marks the deepest visible corner, which seeds an empty box. VGGT fits a Manhattan frame (three orthogonal wall axes) from the point cloud; a closed-form similarity then drops the camera into that box. Room height is the only external metric.
Detection proceeds in three layers: wall-mounted items, free-standing furniture and ceiling fixtures, then decorations that rest on furniture. Half-hidden objects are inpainted, then meshed. Because the inpainted crop still carries the photo's perspective, the mesh arrives tilted. PCA stands it up; the VLM labels facing from multi-view renders so later prompts can say "chair faces the desk."
Placement is depth-first, walking from the far corner along the two walls toward the camera. When a new object is dropped, everything behind it is already there and the foreground is still empty, which keeps the VLM from reasoning through clutter. Each object has a position, a uniform scale, and a yaw taken from a relation phrase such as "sofa back against the left wall."
Those poses are semantic. Geometry refinement has two coupled steps: the rendered silhouette is aligned to the segmentation mask, which fixes lateral shift and the scale-to-depth ratio; VGGT's front-surface median then corrects depth, which resolves metric scale. Collisions move the object with the smaller escape; if that fails, the VLM rewrites placement order. After every layer, a render of the partial scene goes back to the VLM against the photo to catch misses, fix facing, and equalize matched sets (a pair of chairs should share a size) before the next layer starts.
A last VLM pass assigns PBR materials, flags lamps and windows, and retunes emission against a Cycles render.
One scene takes 20 to 90 minutes on an L40, at 3.4 VLM calls per object on average.
Two test sets: 100 Front3D renders with geometric ground truth, and 30 rights-cleared real photos (HARMONY30). Baselines are Gen3DSR, 3D-ReGen, CAST, SAM3D fed HARMONY's own masks (SAM3D), concurrent VIGA, and GPT-6 Astra. CAST has no official release, so an unofficial build is used; CAST and SAM3D do not rebuild the room shell, and perceptual scores give them HARMONY's background.
HARMONY leads every Front3D column:
| Method | N-CLIP↓ | PL↓ | CD↓ | [email protected]↑ |
| SAM3D | 0.127 | 0.069 | 0.056 | 81.79 |
| Gen3DSR | 0.168 | 0.043 | 0.057 | 78.33 |
| CAST | 0.149 | 0.077 | 0.052 | 85.70 |
| 3D-ReGen | 0.163 | 0.142 | 0.060 | 78.87 |
| VIGA | 0.179 | 0.079 | 0.055 | 83.90 |
| GPT-6 Astra | 0.095 | 0.052 | 0.061 | 77.44 |
| HARMONY | 0.092 | 0.041 | 0.049 | 89.82 |
Against Astra the semantic gap is 0.003; geometry is the real split, [email protected] 89.82 vs 77.44 and Chamfer 0.049 vs 0.061. The paper notes Astra looks plausible while beds often miss the wall.
On HARMONY30, N-CLIP is 0.112 and photometric error 0.045, again the best in the table; Astra sits at 0.127 / 0.047 and was left out of the user study. Sixteen people ranked six reconstructions on 20 scenes. HARMONY's mean rank is 1.54, first in 67.0% of choices and top-two in 86.0%. VIGA's mean rank is 4.37, with a 0.7% top-1 share.
The ablation that hurts most is dropping camera calibration (N-CLIP 0.1034 to 0.1381), then killing the per-stage feedback loop (0.1184). Removing depth-first traversal or VGGT snap costs less. Accuracy comes from pinning the camera, then snapping to the image, not from any single module.
This is an untrained assembly line: relations to the VLM, metres to the point cloud. For anyone who needs an editable indoor asset from one photo, the recipe is clear. Language handles "against / on / facing"; geometry handles where and how large; staged critique stops errors from cascading.
It is not a new scene generator. A scene still costs close to an hour, far from interactive. The Astra comparison is the useful one: an end-to-end model can already get the semantics close. Compositional reconstruction still has to win on wall alignment, facing, and metric layout.
The appendix drops finished rooms into rigid-body earthquake sims and into Isaac Sim, where a Unitree H1 and a Franka pull a chair. Those robot paths are scripted. They test contact geometry, not a control policy.
Two failure modes are written up. Rare object types confuse the VLM and the detector vocabularies: 25.9% of detections have grounding confidence below 0.35, while true placement failures after routing sit at 0.3% of objects. A foreground object clipped by the camera keeps the right semantic relation, but the mesh is underconstrained from the render view. The text promises three common failures; the third is never described.
Several scoring choices need a discount. SAM3D borrows HARMONY masks. CAST is unofficial. Methods without a background inherit HARMONY's walls and floor. The user study used 16 unpaid volunteers and did not include Astra. The method assumes Manhattan interiors; outdoor scenes and curved rooms are out of scope. The authors flag multi-view input and object articulation as future work.