Test-Time Spatial Reasoning for Robot Manipulation Using Generative Real-to-Sim
Ivan Kapelyukh, Yafei Hu, Ran Gong, Brandon May, Tushar Kusnur, Laura Herlant, Karl Schmeckpeper, Edward Johns, Xiaohan Zhang
IROS 2026
cs.RO, cs.CV
2026-09-28
From one RGB-D view Simify completes meshes and evolves placements in 1024 sims (~73s). Ablation mean 87.5 vs 10.3 random; TSDF-EA failed 10/10 real stacking trials at planning.
Building the tallest tower, the longest cantilever, or a chain of objects that knocks over a bottle needs precise contact and physical stability. Getting the semantic layout right is not enough. Two common routes stall here.
3D-aware foundation models describe object relations but do not output continuous poses that obey dynamics. End-to-end vision-language-action (VLA) policies, which map images and language straight to actions, need large teleoperation datasets and at test time mostly imitate motions close to training. Unseen objects or new initial poses can change feasible order, orientation, and contacts. Imitation does not invent that plan.
Simify assumes no CAD models, no dataset of successful arrangements, and no learned goal-state predictor. One RGB-D frame becomes a simulation-ready twin. A physics engine then searches placement order and 6D goal poses. The task is a scalar reward, for example the height of the highest point after the scene settles. Writing that reward from language is left open.
Three stages, no training.
Single-view real-to-sim: build a simulatable scene from a real observation. A wrist camera takes one RGB-D image. SAM-2 proposes segments; GPT-4o lists manipulable objects and assigns masks. Image-to-3D models such as Hunyuan3D-2 train on unoccluded views, so each crop first goes through Stable-Diffusion-based amodal inpainting to fill hidden appearance, then mesh generation. The mesh has no metric scale. ICP aligns it to the observed point cloud, Foundation Pose estimates the initial pose, and assets load into Isaac Lab for GPU-parallel simulation. Unobserved faces such as the bottom of a tin can are filled by the generative prior, which later decides whether the can rolls like the real one.
Search uses an evolutionary algorithm on a mixed space: discrete placement order plus continuous 6D poses. A population of random solutions is evaluated in parallel: place in order, wait to settle, score the reward. The top 5% remain as elites; the rest are sampled with a reward softmax and mutated. Gaussian placement noise during optimization pushes solutions toward robustness. After the last generation, each candidate is scored on multiple noisy rollouts, and the highest expected reward goes to the robot. Default setting: 1024 parallel environments, 73 seconds on average. Random poses and orders on the same generated meshes yield a mean score of 10.3. Complete geometry is not a plan.
On hardware, Foundation Pose re-estimates current poses, grasps are sampled on the generated mesh, and cuRobo runs thousands of parallel collision checks. For stacking, the last stretch of the place motion uses the wrist force sensor and lowers until contact, so an open-loop trajectory does not release early.
Each method is run 10 times per task in simulation. Each solution is then evaluated with 2048 noisy placements. Rewards are rescaled to a task score. LLM-GROP uses GPT-5 for zero-shot goal poses, prompted with the reward, axis constraints, a frame-annotated image, and object names, masses, and 3D boxes. TSDF-EA runs the same evolutionary search on partial TSDF meshes (truncated signed distance fields that fuse visible depth and cannot complete hidden faces), ablating the 3D generative prior.
Figure 3: Simify scores higher than both baselines on all three tasks. LLM-GROP is competitive on stacking and unreliable on cantilever and dominoes. The VLM knows a tower should be tall; it does not pin down continuous contact poses. TSDF-EA lags across the board because hidden surfaces are missing.
Ablation of the search (task score, %):
| Method | Stacking | Cantilever | Dominoes | Mean |
| MeshGen-Rand | 29.8 | 0.0 | 1.1 | 10.3 |
| Simify-NoVal | 62.6 | 45.6 | 83.8 | 64.0 |
| Simify | 82.2 | 85.9 | 94.3 | 87.5 |
Dropping the noisy validation stage cuts the mean from 87.5 to 64.0, and cantilever from 85.9 to 45.6. On dominoes, the highest optimization score was a brittle trick: perch the green bottle on the can to tip it. It sometimes worked. Validation picked a higher expected-reward plan that seats the bottle as a spacer and rolls the can down a book ramp.
Hardware: 7-DoF Franka Panda, wrist RealSense D435, 10 trials per task. Simify's mean task progress beats TSDF-EA on all three. TSDF-EA failed all 10 stacking trials at Stage 1, where planning aborts if simulated success sits below a fixed threshold. Partial clouds often never find a stable, executable arrangement. Simify fails planning less often and reaches later placement stages. Figure 5 breaks down 30 real trials; the paper does not report per-category percentages in the text.
Against YCB ground truth, low-occlusion F1 is 0.927 for generated meshes vs 0.682 for TSDF; under high occlusion the pair is 0.825 vs 0.579. TSDF is slightly more accurate on visible surfaces and much less complete. Adding parallel environments raises scores on all three tasks, most clearly on cantilever and dominoes.
This spends test-time compute on physics rather than collecting demonstrators for a VLA. When objects are new, initial poses are new, and nobody logged successful arrangements, one RGB-D frame and a reward function are the interface. Hunyuan3D-2 is swappable.
The niche is narrow and physically hard: multi-object rearrangement where success is stability, balance, or dynamic collisions. Stacking, cantilever, and dominoes fit. Semantic goals such as placing a cup to the right of a plate may already be in reach for LLM-GROP.
The cost is a default 1024 GPU environments, mass and friction treated as known, and hand-written rewards. Relative to imitation, the saving is demonstrations and offline training. The spend is parallel simulation at inference and whatever error the generated mesh carries.
Mass and friction are given; center of mass assumes uniform density; cantilever is sensitive to those numbers. During search, objects teleport to goal poses with no simulated arm. Heuristics drop obviously unreachable rotations, but the plan may still be infeasible on hardware. Single-view image-to-3D precision is the system ceiling. Rigid bodies only. Rewards are not generated from language. The camera viewpoint is given.
Simulation scoring uses the same mesh the method optimized against, so Figure 3 is not a sim-to-real number. Real-robot comparison is only against TSDF-EA; LLM-GROP never ran on the Franka, so the hardware case for beating foundation-model spatial reasoning is incomplete. Three tasks, a small object set, and the cantilever support box is weighted and taped for high friction so a single arm can even attempt it. The 30-trial failure chart has no numeric breakdown in the text. In a home, treating physics parameters as given still means someone has to set them.