LDA-1B: Scaling Latent Dynamics Action Model via Universal Embodied Data Ingestion
Jiangran Lyu, Kai Liu, Xuheng Zhang, Haoran Liao, Yusen Feng, Wenxuan Zhu, Tingrui Shen, Jiayi Chen, Jiazhao Zhang, Yifei Dong, Wenbo Cui, Senmao Qi, Shuo Wang, Yixin Zheng, Mi Yan, Xuesong Shi, Haoran Li, Dongbin Zhao, Ming-Yu Liu, Zhizheng Zhang, Li Yi, Yizhou Wang, He Wang
RSS 2026
cs.RO
2026-02-13
LDA-1B jointly trains policy and dynamics in DINO space on 30k hours of mixed data, hitting 55.4% vs 51.3% same-data GR00T; noisy trajectories add 10 points.
Robot foundation models mostly scale by cloning expert actions. Suboptimal teleop, retries, and unlabeled human video get filtered out. Those traces still carry contact physics: how objects move, how force transfers, what the scene looks like after a miss. Throwing them away discards dynamics priors and leaves a narrow, expensive expert-only pipeline.
Unified World Models (UWM) can in principle train on that mix by jointly modeling policy, forward dynamics, inverse dynamics, and visual forecasting inside one diffusion model. Current systems treat every trajectory the same, so noisy actions leak into the policy. They also predict future frames in pixel or VAE space, where lighting and texture swamp the contact signal. A billion parameters do not help if the latent space cannot separate appearance from interaction.
LDA-1B assigns roles by data quality and moves the prediction target into a structured DINO latent space.
Four learned task embeddings are added to the diffusion timestep so the network knows which problem it is solving. Missing modalities are filled with register tokens: a visual placeholder when training policy, an action placeholder when training visual forecasting. The objective is flow matching. Action and visual losses switch on per task.
Future visuals are DINOv3-ViT-s features at 14×14×384, not pixels. Background clutter is suppressed, so the model has to track where objects go after an action. Actions live in a hand-centric frame: delta wrist pose plus gripper width or wrist-relative finger keypoints. Vision is sampled at 3 Hz, actions at 10 Hz, which cuts redundant nearby frames.
The backbone is MM-DiT. Action and visual experts keep their own QKV and FFN stacks, share self-attention to align the two asynchronous streams, and cross-attend to tokens from a frozen Qwen3-VL-4B. Pretraining updates about 1B parameters in the MM-DiT and action heads. DINO stays frozen. Training ran on 48 H800 GPUs for 400k steps, 4,608 GPU-hours. Finetuning unfreezes the VLM and keeps the mixed-quality recipe.
EI-30k supplies the data: 8,030 hours of real robots, 8,600 hours of simulation, 7,200 hours of action-labeled humans, and 10k hours of actionless egocentric video. Everything is converted to LeRobot format with hand-aligned end-effector frames.
On RoboCasa-GR1 (24 kitchen tasks, 1,000 finetune trajectories each, 51 trials):
| Method | Visual target | Success |
| GR00T-N1.6 (3B) | no explicit dynamics | 47.6% |
| GR00T-EI10k (1B, high-quality subset of the same data) | no explicit dynamics | 51.3% |
| UWM + MM-DiT | VAE | 20.0% |
| LDA (plain DiT) | DINO | 48.9% |
| LDA-1B | DINO | 55.4% |
Swapping VAE for DINO jumps success from 20.0% to 55.4%. Dropping MM-DiT costs 6.5 points; shrinking to 0.5B costs 4.7. Data and size explain a slice. The latent space explains more.
On Galbot G1, held out from pretraining, few-shot pick-and-place lands at 80%–90%. Flip-the-box is 60% against GR00T at 20%. Watering is 80% against π0.5 at 60%. Long-horizon trash dumping is 35% against 0 for both baselines. On dexterous hands, pull-nail is 80% (π0.5 near 0, GR00T 40%) and flip-bread is 90% (both baselines 10%). The paper’s category ceilings versus π0.5 are +21% contact-rich, +48% dexterous, +23% long-horizon.
Under visual and spatial shifts on pick-and-place, LDA holds 60% on novel objects and backgrounds (GR00T 40%, π0.5 about 27% / 20%) and 40% on OOD start poses (20% and 6.7%).
Mixed-quality finetuning is the sharper result. Place-pen-in-box: LDA goes 70% to 80% after adding low-quality traces; π0.5 falls 60% to 40%. Bimanual lid removal: LDA 50% to 60%, π0.5 50% to 40%. About 30% of the trajectories that BC pipelines usually discard are worth 10 points under this objective split.
On a held-out Agibot World split, action-prediction L1 falls to 6.6 at 30k hours. Policy-only training degrades when noisy data is added. With all four objectives on, another 10k hours of actionless video still cuts error.
This is a usable recipe for robot pretraining: keep the messy traces, but put them on dynamics losses. Pixel-space world models look saturated near a billion parameters; a semantic latent target is a bigger lever than another DiT stack. On the deployment side, teleop no longer needs an expert-only filter, which lowers collection cost. Galbot G1 was absent from pretraining and still few-shot transfers, so the hand-centric action space at least spans these platforms.
It is still an engineering assembly, not a new formulation. The four UWM objectives already existed. The work is role-aware data, DINO latents, MM-DiT, and the grind of aligning 30k hours. Real-robot trials are about 10–20 per task, so the percentages will move.
The paper flags frozen DINO features and almost entirely egocentric cameras. New viewpoints or extra sensors may not transfer.
A few claims sit on thin ice. Real-robot baselines finetune on the filtered expert subset; LDA sees all 100 traces, so “noisy data helps” is entangled with “30% more data helps.” Coordinate frames are aligned by hand, which will not scale to the next 10k hours. Figure 1 says 1.6B parameters; the tables and title say 1B trainable. Trash dumping at 35% beats 0 and is still far from usable. The scaling proxy is action L1 on Agibot World, one step removed from real success rates.