Supervise What Survives: Geometry-Guided VLA Adaptation from Synthetic Robot Videos
Danze Chen, Yanzhe Chen, Qiming Huang, Zhijun Cao, Chen Gao, Mike Zheng Shou
cs.RO
2026-06-23
GRA supervises the VLA vision backbone with 2D waypoints from human video and trains actions on 25 real demos, hitting 68.9% success vs 48.9–54.4% for pseudo-action baselines.
VLA models need paired video and motor commands. Real teleoperation is expensive: a Franka arm, trained operators, and time on the hardware. Human manipulation videos are cheap. Human-to-robot video generators can now paint robot execution at scale. This paper uses Wan 2.2, fine-tuned with Mitty's LoRA recipe. The generator produces pixels. It does not produce physics.
The default move treats those pixels as demos that lost their action labels. DreamGen-style pipelines recover pseudo-actions with an inverse dynamics model. MimicDreamer-style pipelines retarget geometry into delta commands. Both dump the result into the VLA action head. Video models optimize for visual plausibility, not control laws. Mapping hallucinated pixels onto rigid motor commands teaches the action head a distorted controller.
The paper's rule is simpler. A generated video keeps geometry: where the end-effector goes, how objects sit. Control, the exact motor commands, does not survive generation. Supervise what survives.
GRA, Geometry-guided Representation Alignment, sits on OpenVLA-OFT. A vision backbone maps images into Llama-2 7B embedding space. A continuous L1 head outputs 24-step 7-DoF delta actions: translation, roll-pitch-yaw, gripper. Geometry takes a separate path.
2D waypoints are computed from the source human video, never from generated pixels:
Generated robot frames and waypoint labels share frame indices from the human source. They do not share a pixel path. That decoupling is the design.
Training has two stages. Stage 1 runs 5K steps: LoRA rank 32 on the vision backbone plus a 3-layer MLP 2D head, regressing future waypoints on synthetic frames. The language model and action head stay out of the forward pass. Stage 2 runs 10K steps on real teleoperation only. The action head sees 25 real demos. The Stage-1 2D head stays on as a spatial anchor, λ=0.5, using proprioceptive end-effector projections so action gradients do not wash out spatial structure. One H200, batch 16, learning rate 5×10⁻⁴.
Hardware is a Franka Research 3 with a fixed RealSense D435. Three tabletop pick-and-place tasks, 30 trials each: cube to pad, cup to coaster, mango to plate. Matched budget is 25 real trajectories plus 75 generated videos per task. The upper reference is 100 real trajectories and no synthetic data. Success is experimenter-judged: object on the target with gripper open. Slip-offs and early gripper opens count as failures. Control runs at 10 Hz with a 30 s timeout.
| Method | Cube→Pad | Cup→Coaster | Mango→Plate | Mean |
| Real-only (100) | 73.3% | 63.3% | 90.0% | 75.6% |
| Real-only (25) | 60.0% | 46.7% | 76.7% | 61.1% |
| DreamGen-style | 46.7% | 36.7% | 63.3% | 48.9% |
| MimicDreamer-style | 53.3% | 40.0% | 70.0% | 54.4% |
| GRA | 66.7% | 56.7% | 83.3% | 68.9% |
GRA is 7.8 points above matched-budget Real-only. The gap to the 100-demo reference shrinks from 14.5 to 6.7 points on a quarter of the real data. Both pseudo-action baselines fall below Real-only: DreamGen-style by 12.2 points, MimicDreamer-style by 6.7. All three tasks move the same way.
Probes pin down the asymmetry. On off-the-shelf SigLIP, in-domain waypoint error is 0.14σ versus 0.63σ for delta actions, about 4× easier to read out. Train on synthetic, test on real: waypoints 0.41σ, delta actions 1.38σ. Stage-1 waypoint pretraining drops in-domain waypoint error to 0.10σ and raises delta-action error to 0.82σ. The supervision route changes what remains readable in the backbone.
Teacher-forcing per-step error, feeding ground-truth observations: GRA 9.26 mm position / 0.060 total L1; Real-only 11.09 mm / 0.090; MimicDreamer-style 13.36 mm. Spatial readout L1 ties GRA and MimicDreamer-style at 0.020, yet MimicDreamer trails 14.5 points in closed loop. A spatially readable backbone is not a clean controller.
Ablation on the hardest task, cup to coaster: full GRA 56.7%; drop Stage 1, 43.3%; drop the Stage-2 anchor, 46.7%; swap waypoints for retargeted delta actions, 36.7%, worse than Real-only at 46.7%. All three pieces contribute. Changing the supervision content hurts more than changing the route.
Synthetic robot video is becoming the default data patch for VLAs. The default pipeline generates frames, recovers actions, and treats them as demos. This paper is a counterexample: in this setting, pseudo-actions pull the policy down by more than ten points.
The usable recipe is narrow and concrete. A small pile of real teleoperation (25 demos per task here) plus human-to-robot generated video. Geometry goes to the vision backbone. Control stays on real data. One H200 covers both stages.
This is a routing fix, not a better generator or a new backbone. Closed-loop performance still tracks the scale of real action data. The authors say so.
The authors flag both ends. Upstream, the spatial-control split is an empirical observation, not a characterization across generators, and they do not test whether anything beyond geometry survives. Downstream, closed-loop success still scales with real action data; raising that ceiling needs generators or action decoders that produce trustworthy control.
The experiment is thin. Three tabletop pick-and-place tasks, one Franka, one fixed third-person camera, 30 trials per task, success scored by the same experimenter. Visual judgment does not reproduce. Pseudo-action baselines are style reimplementations, not official ports. The DreamGen-style IDM is trained on real demos and then run on generated frames, so domain gap is baked into the labels.
The waypoint pipeline itself is noisy. Held-out Stage-1 MAE is 18.5–23.2 pixels, 3.8–4.8% of image width. Geometric supervision is not a clean label. It is more stable than recovering control from generated pixels.