SLIP-VLA: Single-Step Latent Imagination for Policy Learning in Vision-Language-Action Models
Tianfu Li, Haoxuan Xu, Wenbo Chen, Haitian Li, Changchuan Yang, Xinhu Zheng, Jun Ma, Yuan Liu, Lujia Wang, Haoang Li
cs.RO
2026-09-27
SLIP-VLA produces dense future latents in one denoising step, shapes them for geometry, semantics, and control, and hits 83.2% on LIBERO-Plus at 12 ms imagination.
Most Vision-Language-Action (VLA) policies map current camera frames and a language command straight to actions. Grasping, alignment, and insertion depend on where contacts and object poses will sit a few steps later, so reading only the current frame leaves the policy guessing.
Recent future-aware VLAs first predict an image or a motion cue, then decode actions. Those cues are usually sparse subgoals from slow autoregressive decoding, and they do not cover the full action horizon. World Action Models (WAMs) fill in temporally dense visual dynamics, at the cost of iterative denoising. One-step variants cut latency and typically lose to the multi-step versions. The question is how to shape the latent from a single denoising update so a policy can actually control a robot with it.
SLIP-VLA, from HKUST (Guangzhou), HKUST, NTU, and Zhejiang University, stacks a Qwen3.5-2B VLM, a 12-layer World DiT, and a 24-layer Action DiT. The VLM encodes the current multi-view images and the instruction. The World DiT then runs one denoising pass over K=4 action-aligned future slots and injects selected hidden states into the Action DiT by layer-wise cross-attention. Future RGB is never decoded at test time. VGGT-Ω, DINOv3, and the dynamics heads exist only during training.
The imagination source is anchored on the current observation. The current visual latent is repeated across the four slots and mixed with same-scale Gaussian noise at ratio η=0.2. Pure noise would wipe scene layout in one step; a clean current latent would leave too little room for change. Intermediate World DiT layers feed the Action DiT. The last layer is not the only source.
Two training-time constraints shape those tokens.
Perceptual sufficiency. An earlier layer is aligned with frozen VGGT-Ω geometric features, computed over the current frame plus the K future frames as one temporal window, matching cosine direction and feature magnitude. A deeper layer is aligned with frozen DINOv3 patch features, one future frame at a time. Geometry localizes future contacts. Semantics marks which object and which part matter.
Control sufficiency. A lightweight latent world model predicts the next slot from the current slot plus the intervening action chunk, with a ranking term so the true action fits better than a shuffled one. An inverse-dynamics head recovers that chunk from the two consecutive latents, using mismatched-next and same-state negatives so it cannot cheat from a single frame. Motion uses Smooth-L1. Gripper state uses binary cross-entropy.
Training jointly optimizes conditional flow matching on 16-step action chunks, video prediction on a random-timestep branch and a first-step branch, plus the perceptual and control losses. The first-step timestep is annealed from near the source endpoint down to zero, matching inference. Optimization is staged: stabilize the predictive latent, then train the full policy. At inference the pipeline is one World DiT forward plus Action DiT decoding. Imagination takes 12 ms. End-to-end latency on LIBERO-Plus is 181 ms. Training uses eight A100 GPUs.
Three simulation suites and one real robot, against generalist VLAs, future-aware VLAs, and WAMs.
| Method | LIBERO | LIBERO-Plus | RoboTwin 2.0 |
| π0 | 94.4 | 53.6 | 62.2 |
| LaMP | 98.3 | 79.3 | / |
| Fast-WAM | 97.6 | 49.1 | 91.8 |
| Fast-WAM-Joint | 98.5 | 68.7 | 90.6 |
| SLIP-VLA | 98.9 | 83.2 | 91.8 |
On LIBERO, SLIP-VLA averages 98.9, with Spatial 99.6, Object 99.8, Goal 98.2, and Long 98.0. That is 0.4 points above Fast-WAM-Joint. The in-distribution set is already near ceiling, so 0.4 points does not change a deployment decision.
The gap that matters is zero-shot shift. On LIBERO-Plus, SLIP-VLA averages 83.2, 3.9 points above the strongest baseline LaMP at 79.3, and 14.4 points above the best WAM, Fast-WAM-IDM at 68.8. On RoboTwin 2.0, 50 bimanual tasks, it averages 91.8, tying Fast-WAM, with no extra robot pretraining data.
Ablations line up. A naive single-step run scores 97.7 / 79.0 on LIBERO / LIBERO-Plus. Perceptual shaping alone reaches 98.3 / 81.4, control shaping 98.5 / 82.3, both 98.9 / 83.2. The out-of-distribution lift is larger. Linear probes on frozen latents: geometric Pearson rises from 0.804 to 0.916 and AbsRel drops from 0.188 to 0.112; object-mask mIoU rises from 0.388 to 0.590. Current-latent plus a little noise beats pure Gaussian on LIBERO-Plus, 83.2 versus 82.5.
Raising denoising from 1 step to 4 does not steadily raise PSNR, and LIBERO-Plus success peaks at one step. After shaping, extra denoising is wasted compute in this setup.
Real-robot tests run on an AgileX COBOT Magic bimanual platform, 50 trials per condition, compared only with π0. Corn-to-plate rises from 72% to 86%. Unseen background, appearance, and orientation score 70 / 70 / 66 against π0 at 58 / 54 / 52. The two-step open-lid-then-place task hits 82% (π0 68), the three-step add-close-lid task 74% (π0 60). Absolute gains sit between 12 and 16 points.
For teams already training a DiT-style VLA with a video head, the transferable claim is narrow and useful: the future does not need to be rendered as pixels, and it does not need iterative denoising. One forward pass yields a dense latent. Geometry and semantics teachers plus forward/inverse dynamics shape it at train time. All of those modules come off at inference.
This is incremental. The 0.4-point LIBERO bump is noise for practitioners. The number worth acting on is the LIBERO-Plus margin against WAMs, plus dropping iterative imagination. End-to-end latency is still 181 ms. Action DiT remains the bottleneck. The 12 ms figure is imagination only.
The recipe needs future frames in the demonstrations so VGGT-Ω and DINOv3 targets can be extracted. Without those frames, the shaping losses have nowhere to sit.
The paper has no Limitations section. The conclusion only says a shaped predictive latent does not need a fully denoised video.
Real-robot comparisons are against π0 only, not LaMP or Fast-WAM. Whatever edge showed up in simulation is untested on hardware. The task family is narrow: corn-to-plate plus open-pot variants, one platform.
The backbone is Qwen3.5-2B. Baselines in the tables do not share one VLM, so a top-line ranking is not a matched-backbone comparison. Training still runs VGGT-Ω and DINOv3 on eight A100s; that cost sits on the training side. Horizon is K=4 slots and 16-step chunks. Longer horizons are untested.
In-distribution 98.9 falling to 83.2 on LIBERO-Plus means distribution shift is only partly absorbed.