DyPES-VLA: Learning Shared Dynamics Priors and Embodiment-Specific Control for Cross-Embodiment Manipulation
Junfeng Li, Junjie He, Zhide Zhong, Yangyang Zheng, Pingyue Sheng, Jiayu Dong, Ruixin Li, Haodong Yan, Jiaguan Zhu, Tianran Zhang, Runze Yu, Wen Chen, Liuqing Yang, Yuxiang Gao, Haoang Li
cs.RO
2026-08-07
DyPES-VLA avoids forcing heterogeneous robots into one action format; it learns shared dynamics priors via future-frame prediction and decodes them with per-embodiment MoE heads, reaching 98.0/89.02/59.25 percent from a single checkpoint.
Training one VLA (vision-language-action) policy that works across different robots is stuck on two fronts. First, existing methods supervise only with action labels, wasting the recurring patterns of object motion, contact, and scene change in large collections of human and robot videos. Second, to fit different robots' actions into one format requires heavy manual preprocessing (coordinate transforms, inverse kinematics) that scales poorly across morphologies. Worse, this forced alignment entangles two things that should stay apart: interaction regularities shared across embodiments and control semantics unique to each robot. The HKUST-GZ team separates them.
DyPES-VLA uses a set of learnable query tokens as the shared interface. The VLM encodes vision, instruction, embodiment metadata (robot model, control frequency, action dimension), and these queries together, producing query states. Two supervision paths are the key. The first is a future generation head that forces the query states to predict a future frame; since the information needed to synthesize it must pass through the query tokens, they are driven to capture shared dynamics priors, and this step needs no action labels, so it can consume action-free video. The second is an action head, a flow-matching DiT (diffusion transformer). Its attention layers are shared across embodiments to capture temporal action structure, while its feed-forward networks are K embodiment-specific experts selected by a static router from metadata, generating action chunks directly in each native action space with no pre-alignment. Training is two-stage: pretrain the queries and future head on action-free video, then jointly train both heads on action-labeled multi-embodiment demos; the future head is dropped at inference.
| Benchmark | Embodiment | DyPES-VLA | Main rivals |
| LIBERO | 7-DoF single-arm | 98.0% | X-VLA 98.1, Fast-WAM 97.6 |
| RoboTwin 2.0 | 14-DoF dual-arm | 89.02% | Qwen-VLA 86.65, ABot-M0 85.5 |
| RoboCasa-GR1 | 29-DoF humanoid | 59.25% | ABot-M0 58.3, LDA-1B 55.4 |
A single checkpoint reaches state of the art on all three, though some leads are narrow (LIBERO is only 0.4 above Fast-WAM and 0.1 below X-VLA; RoboCasa is 0.95 above ABot-M0). The real-world result is more convincing: the same checkpoint, fine-tuned on FR3 single-arm, COBOT Magic dual-arm, and G1 humanoid, averages 75.6% across three tasks, 16 points above GR00T-N1.6's 59.6% and far ahead of ACT's 32.4%. In ablations, removing future prediction hurts most (RoboTwin down 2.4, RoboCasa down 2.5); a linear probe confirms that with future supervision the query states linearly decode future contact onset and release (AUROC up across the board).
It charts a clean path for cross-embodiment robot training: share what is shared (learn interaction regularities from action-free video and future prediction) and specialize what is specific (decode in native action spaces via MoE experts), with no manual action-format unification. For generalist-robot teams this means adding a new embodiment may take just a new expert set and its demos rather than redoing alignment. The 75.6% real-world average across three morphologies shows the shared priors transfer to physical control of different bodies.
The VLM backbone is only Qwen3-VL-2B, on the small side, and the paper does not test the ceiling a larger backbone might bring. Several simulation leads are fractions of a point and may not be statistically robust. Real-world testing covers only three tabletop tasks (place a kiwi, pour water, place a book), all relatively simple pick-and-place, with no harder assembly or long-horizon tasks. The paper never discusses failure modes. The ablations do not isolate "cross-embodiment" as a variable, so it is unclear how much of the shared prior comes from joint cross-embodiment training versus from future prediction within a single embodiment.