SoftVTBench: A Deformation-Aware Visuo-Tactile Dataset and Benchmark for Deformable-Object Manipulation
Bowen Jing, Mingxin Wang, Ruiyang Hao, Chenchen Ge, Hanwen Shen, Junjie He, Yang Cui, Yiming Hou, Weitao Zhou, Jiawei Wang, Minglei Li, Dandan Zhang, Ding Zhao, Houde Liu, Xiaofan Li, Si Liu, Ping Luo, Haibao Yu
cs.RO
2026-08-19
SoftVTBench adds DSR to 4,000 visuo-tactile demos. In all 12 ID setups, 0.7–24% of successes exceed squeeze tolerance. Touch helps more under shift than in-distribution.
Manipulation suites mostly score task completion. For deformable objects that is the wrong summary. A loose grasp slips; a tight one crushes the roll. Both can look like “placed” in a success bit. Vision also loses the contact interface once the fingers close. Touch sees geometry, shear, and local compression directly.
The missing piece is a record with two streams: policy-visible visuo-tactile observations, and evaluator-only physical state. Prior benchmarks split the four requirements: complete tasks, volumetric soft bodies, policy-visible touch, hidden deformation used in the score. SoftVTBench ships all four.
Data are collected in Isaac Sim / Isaac Lab: 4,000 scripted expert demos, 40 pick-and-place tasks, four suites (object-soft, spatial-soft, and matched rigid twins). Each suite has 10 tasks and 1,000 demos. At 20 Hz the log aligns third-person and wrist RGB, dual-finger tactile RGB and marker motion, proprioception, language, and both binary and continuous gripper commands. Touch is rendered with TacEx’s GelSight Mini profile. FEM nodes are stored for the evaluator and never shown to the policy.
Deformation is rigid-motion-removed nodal RMS displacement, normalized by the initial bounding-box diagonal. Before any policy is trained, a scripted grasp sweep sets an object tolerance τ at the 90th percentile of stable grasps. DSR counts episodes that finish the task and keep peak normalized deformation ≤ 1. TSR is kinematic completion; TSR minus DSR is exactly the share of “placed but crushed” rollouts.
In-distribution eval uses 500 held-out episodes per suite. Out-of-distribution shifts one factor at a time: light, mass, or Young’s modulus, nine conditions. Policies are Diffusion Policy, π0.5, and FastWAM, each in vision-only and visuo-tactile variants.
DSR is below TSR in all 12 in-distribution deformable configurations. Diffusion Policy visuo-tactile on Object-Soft is 40.0% TSR and 30.4% DSR, a 9.6-point gap, 24% of that configuration’s successes. FastWAM’s violation share is 0.7–6.5%, and on spatial suites TSR and DSR differ by at most 0.4 points.
| Setup | TSR | DSR |
| Diffusion Policy VT-C Object-Soft | 40.0% | 30.4% |
| π0.5 VO-C Object-Soft | 41.6% | 38.4% |
| FastWAM VT-C Spatial-Soft | 56.4% | 56.0% |
Rigid twins show that deformability is not a flat tax. π0.5 drops from 60.0% rigid to 41.6% soft on the object axis; Diffusion Policy and FastWAM drop 2.6 and 2.0 points, inside protocol resolution. On the spatial axis FastWAM scores 12.0 points higher on soft assets.
Sensing and gripper granularity have to be crossed. For π0.5 on Object-Soft, continuous control alone lifts TSR from 30.2% (VO-B) to 41.6%; touch alone (VT-B) reaches 41.0%; both together 41.4%. VO-C and VT-B differ by 0.6 TSR points and by 10.4 DSR points (38.4% vs 28.0%). Comparing VO-B with VT-C would credit touch for an actuation gain.
Under shift, visuo-tactile wins TSR in all six policy–suite pairs and DSR in five. π0.5 visuo-tactile barely moves on Object-Soft (41.4%→41.0% TSR); vision-only falls from 41.6% to 35.8%. In-distribution tactile gains are mixed; under shift the direction is steadier.
Completion and contact quality are different rankings. Food, soft packaging, and medical materials cannot treat TSR as safety. The matched controls stop a lazy “touch helped” story: continuous gripper commands can reproduce the same TSR lift. Tactile fusion is still unsolved in-distribution. FastWAM shows completion and shape-keeping can align; that tradeoff is not a law of soft objects.
All of it is simulated. The appendix states that published Taxim/FOTS real-sensor checks do not transfer to these assets; there is no GelSight comparison here. τ is an operational score, not a material-failure threshold. Expert demos stay inside tolerance, so learned violations are not copied from labels, but the expert is scripted. Diffusion Policy is not language-conditioned, so spatial suites that name the target in text confound it. Some rigid visuo-tactile rows decode binary gripper commands from continuous checkpoints and are supporting evidence only.