PhysCaP: Grounding Code-as-Policy Agent with Physics-Informed Exploration
Chen-Yu Lin, Jing-Wen Chen, Hsueh-En Chang, Hung-An Chen, Sheng-Hsun Chang, Chi-Pin Huang, Fu-En Yang, Min-Hung Chen, Yi-Ting Chen, Yu-Chiang Frank Wang, Shao-Hua Sun
cs.RO
2026-08-21
NTU and NVIDIA add training-free mass and stiffness probes to Code-as-Policy. Empty-can success is 8/10 with 2.5 interactions, versus 3.9 for exhaustive probing.
Vision-language-action policies imitate demonstrations well enough to fold cloth and rearrange objects. They eat passive vision. Whether a can is empty, or an avocado ripe, often does not show up on camera. People lift and squeeze. Reinforcement learning can in principle discover that information-seeking, but the sample cost is high, rewards are sparse, and the resulting policy is hard to edit.
Code-as-policy writes perception, reasoning, and control as executable modules, which is easier to inspect than a monolithic policy. Existing CaP agents still stop at passive perception: they plan from what they see, and they will not reach out to measure a missing physical quantity. PhysCaP adds that layer. With no extra tactile hardware, the arm's own sensors estimate mass and stiffness, and a pair of agents decide whom to probe and when to stop and act.
The stack sits on CaP-Agent0. Low-level APIs handle grasp, place, and pose. Object localization uses Molmo2 to point in the image, then a ZED 2i depth map back-projects to 3D. Two training-free PhysX modules supply the physics.
Mass (getmass): run a fixed 15 cm lift empty-handed to log joint torques, repeat while holding the object, and take the difference Δτ as the gravitational contribution. A vertical Jacobian converts torque into a mass estimate that is less sensitive to how the object sits in the gripper.
Stiffness (getstiffness): close the jaws in small steps, run a brief backoff test to reject internal friction, store contact displacement d0, then keep closing until normalized motor effort hits 0.50. Interpolated deformation maps onto a 1 (ultra-soft) to 5 (rigid) scale. Five measurements per object, majority vote.
Exploration is split. The Planner looks at the task and the scene, lists missing physical quantities and candidate objects, and stops once the evidence is enough for the coding agent. The Prioritizer drops implausible actions with visual heuristics (a cup too small to hide the cube; a sealed can is unlikely empty) and ranks the rest. Merging Planner and Prioritizer into one agent (PhysCaP-joint) collapses into unstructured exhaustive plans, which is why they are kept apart.
The default backbone is Gemini 3.1 Pro, shared with every baseline. The robot is a 7-DoF AgileX PiPER. Each real task runs 10 trials.
Vision-only CaP scores 10/10 on finding the blue cube because it lifts all three cups (3 interactions). Empty can is 2/10 and ripe avocado 1/10; the camera cannot see the hidden property. Adding PhysX without selection gets empty can to 7/10 at 3.9 object interactions and 268 s, and avocado to 9/10 at 4 interactions and 516 s. Full PhysCaP cuts the wasted probes:
| Method | Empty-can SR | Interactions | Time |
| CaP (vision only) | 2/10 | 2.5 | 72 s |
| CaP+PhysX | 7/10 | 3.9 | 268 s |
| PhysCaP | 8/10 | 2.5 | 239 s |
On the cube task PhysCaP is 9/10 with 1.33 interactions and 40 s, against 2.7 interactions and 77 s for exhaustive PhysX. Avocado is 9/10, 2 interactions, 300 s, about half the exhaustive probe count. PhysCaP-joint loses success on all three tasks.
The mass module was checked on five calibration weights from 13 g to 963 g, 20 trials each; relative mass gaps are stable. Replacing hardware readings with ground truth (oracle) takes empty-can success to 10/10, so some remaining failures are torque noise. Stiffness was calibrated on a 3D-printed rubber-band button: unripe fruit matches a multi-band hard stop, ripe fruit a softer one.
In a 50-trial LIBERO empty-cup task, PhysCaP reaches 78% success at 1.44 interactions. OpenVLA is 0%, π0.5 4%, MolmoAct2 23%. Those VLAs have no "form a hypothesis, then reach out and measure" loop, so they guess under partial observability.
For people building manipulation stacks, this turns active perception from "train another exploration policy" into "give CaP two measurement APIs and two VLMs that decide whom to touch." No tactile skin is required; joint current and torque already separate empty cans from full ones. The Prioritizer uses shallow common sense: do not weigh sealed cans first, do not squeeze the green avocados first. The gain is fewer wasted interactions, not a leap in measurement accuracy.
This is assembly, not a new dynamics learner. Tasks were deliberately limited to tabletop pick-and-place so the measurement modules could be isolated. Shipping it assumes you already have a tunable CaP stack.
The authors list three: commercial VLM APIs add latency and reasoning jitter; single-camera 2D pointing plus depth sends localization error straight to the end-effector; PiPER communication lag sometimes makes the real trajectory diverge from the generated code.
The evaluation is narrower still. Ten real trials per task, and interaction counts plus time are averaged only on successes, so failed exploration is dropped. Adding the Planner alone drops cube and avocado success relative to CaP+PhysX (10/10 to 9/10, 9/10 to 8/10): the stop rule can fire too early. In simulation, CaP+PhysX+Planner is at 62%, below 74% without the Planner. Empty-can PhysCaP at 8/10 versus 7/10 is a thin gap on n=10. Objects are standard commercial items on a 120×70 cm table; cluttered desks and deformable objects are untested.