DeformSmith: Physics Harness-Guided Hierarchical Generation of Deformable Assets for Robot Manipulation
Can Li, Jie Gu, Zishun Deng, Jingmin Chen, Lei Sun
cs.RO
2026-09-16
DeformSmith builds deformable assets from text or one image. Physical realism is 0.70 vs 0.46 for PhysGen3D; pick-and-place success rises from 40% to 67%.
Robot simulation needs more than a pretty mesh. It needs an object that deforms when gripped and still stands after release. For deformables, geometry, material, and contact are coupled, and a caption or a single image barely constrains Young's modulus or friction. PhysGen3D and PhysGM can infer interactive representations from a still image, but those estimates are initial guesses. Without simulation tests they flatten, tear, or fragment under drop and grasp.
Video methods such as PhysTwin and DeformMaster, and real-robot Real2Sim, need an object that already exists. DeformSmith aims narrower and harder: automatically build volumetric deformable solids from text or one image, excluding fluids and granular media, and make them graspable in simulation.
Four layers, with later layers freezing geometry and the physical model once those are accepted.
A shared physics-grounded harness runs Planner, Designer, and Critic as LLM agents (GPT-5.6 Sol by default). The Prober runs geometry checks or simulation. The Orchestrator enforces a contract: editable fields, bounds, hard gates, per-finger force budget, revision cap. Failed attempts are kept as labeled diagnostics. Particle trajectories can drive Gaussian replay without rerunning physics.
39 cases: 30 text-driven, 9 image-based from PhysGen3D's public assets. GPT-6 Astra, stronger than the harness model, scores physical realism, photorealism, and semantic consistency on 0–1. Forty people do blinded pairwise preference, ties dropped. The shared drop protocol is a 5-second fall from one-quarter object height onto a table, 30 fps, two views.
| Method | Physical realism | Photorealism | Semantic consistency |
| PhysGen3D | 0.46 | 0.34 | 0.80 |
| PhysGM | 0.39 | 0.27 | 0.68 |
| PhysX-Omni | 0.41 | 0.32 | 0.69 |
| DeformSmith | 0.70 | 0.58 | 0.82 |
Physical realism and photorealism each beat the strongest baseline, PhysGen3D, by 0.24; semantic consistency only by 0.02. Human win rates versus PhysGen3D are 68% / 88% / 63% for physical / visual / semantic; versus PhysGM 73% / 95% / 71%; versus PhysX-Omni 75% / 96% / 76%. Qualitatively, baselines flatten, smear, or fragment after the drop; DeformSmith still reads as a turtle, whale, seal, or rugby ball.
Ablations:
| Setting | Metrics | Control → full |
| Hierarchical vs flat | delivery / physics pass | 83% / 80% → 93% / 87% (13.7 vs 13 sim calls) |
| Harness vs one-shot | material target / hard fail | 40% / 17% → 73% / 7% (14 construction calls) |
| Manipulation feedback | task / material / joint success | 40% / 67% / 27% → 67% / 83% / 57% |
The robot ablation uses six assets with five held-out conditions each. The real-world section is a sketch: use the asset as an init for DeformMaster or EMPM. No closed-loop number.
Deformable grasp data is scarce. If generated assets can be squeezed, carried, and released in sim, with replayable particles and contacts, policy learning gets a synthetic path. The practical content of the hierarchy is: fix mass and contact before tuning material, so a wrong modulus cannot hide a wrong mass.
The visual and physical-plausibility gains are the clear story. Parameter accuracy against real materials is not. Semantic consistency barely moves, so "looking like the requested object" was not the main gap.
Coverage is homogeneous volumetric solids. Laminates, cloth, fluids, and granular media are out. Contact is approximate. Parameters inferred from text or images still need real-world checks; the robot photos are a potential application, not a result. Headline metrics are GPT-6 Astra scores and 40-person preference, not Young's modulus error or real grasp success. The L3 ablation is small. The harness uses GPT-5.6 Sol and the judge uses GPT-6 Astra, so the judge is stronger than the generator and may prefer reasonable-looking renders over constitutive truth.