DreamFree drops video generation for robot control: 17.8× faster than DreamZero, 4× π0.5's success

Future Representation Distillation for Imagination-free World Action Models

Jaehyeong Jo, Sangwon Jang, Sihyun Yu, Seonghyeon Ye, Yekyung Nah, Hojung Jung, Kimin Lee, Sung Ju Hwang

cs.RO

2026-11-03

DreamFree distills a frozen video diffusion model into a current-frame-only encoder: 64% real-robot success, 11Hz closed-loop control on a 14B backbone, no video generation.

What problem this solves

World Action Models (WAMs) tap the physics knowledge inside pretrained video diffusion models to drive robots, but the existing recipe actually generates future video at inference: either denoising video and actions jointly, or decoding actions from predicted future frames. Two costs follow. Iterative denoising is slow, so large backbones cannot close the control loop on a real robot. And when the imagined future is wrong, the error flows straight into the actions; most of baseline DreamZero's real-world failures come with visibly off video predictions. The open question is whether action prediction needs video-level detail at all. This paper argues it does not: the predictive knowledge can be compressed into visual features computed from the current frame alone.

Method

DreamFree repurposes a video diffusion backbone as a predictive visual encoder, with representation learning and action learning decoupled into two stages.

Stage one needs no action labels, so any demonstration video works:

• The encoder initializes from a pretrained Wan2.2 (5B or 14B). A structured mask on the backbone's bidirectional self-attention lets current-frame tokens attend only to each other, so the features depend on the current observation and instruction and nothing else.
• A frozen copy of the same video diffusion model plays teacher: it reads a noised video containing the current frame plus real future frames, and one hidden layer's states serve as the distillation target. A lightweight projector maps the encoder's current-frame features onto the teacher's spatiotemporal token layout, and the two are aligned by similarity, a design that follows REPA. Teacher and projector exist only during training.
• The backbone's native flow matching generative loss is kept as a regularizer so pretrained knowledge does not drain away.

Stage two trains the action module on action-labeled robot data: a Perceiver-style latent-query readout distills the encoder features into a fixed number of action-facing tokens, and a 1B DiT decoder conditioned on those tokens, the instruction, and proprioception generates a 32-step action chunk via flow matching. At deployment the encoder runs once, the decoder emits actions, and no video denoising ever happens. That is where 11Hz comes from.

Results

On the real-world DROID setup, evaluated out of the box with public checkpoints (10 tasks, 5 trials each): DreamFree 64% success, DreamZero 56%, π0.5 16%; task progress 83% vs 71.8% vs 33.5%. DreamZero fails every trial on deformable-object and successive-rearrangement tasks such as covering a strawberry with a towel, hanging a hat on a tripod, and putting bolts in a bowl, exactly where DreamFree pulls ahead.

• Latency: about 91ms per action chunk (11Hz) on an H200 with the 14B backbone, versus 1632ms for DreamZero, a 17.8× speedup. Latency is 1.1× π0.5's while success is 4× higher.
• Scaling: going from 5B to 14B lifts RoboCasa by 13.2 points; Fast-WAM under the same scaling moves only from 44.4 to 44.7. Adding unlabeled RoboCasa365 videos pushes the 14B model to 74.4%.
• Ablations: dropping the distillation loss drops RoboCasa to 47.2; dropping the generative loss, to 55.6. Distillation carries the method.
• Probing: from the current frame alone, the encoder's action-prediction error beats the VDM's current-frame features, the no-distillation variant, and even VAE latents of current plus ground-truth future frames, approaching the teacher's error when it sees the real future.

Transfer from videos: composite tasks go from 0% to 21.7%, and the same videos without distillation stay at 0%; videos of a Franka dual-arm setup teach an ALOHA robot new tasks, 6.7% to 30%; after policy training, fine-tuning only the encoder with 10 videos adapts a new task in about 10 minutes on one H200, action module frozen.

Why it matters

The inference-cost problem of video-generating WAMs is dismantled head-on, and since stage one never touches action labels, internet videos and videos from other embodiments can all feed it. The experiments show stronger video backbones and more video data convert directly into higher success, which makes the recipe scaling-friendly; Fast-WAM barely moves when its backbone scales, so distillation fidelity is the operative difference. Closed-loop control at 11Hz is a practical regime on a real robot. One honest caveat: 98.6 vs 98.5 on LIBERO is a tie on a saturated benchmark, and the real gaps show up on RoboCasa's many-tasks-little-data setting and in the real world.

Limitations

The authors' own admission: not every task can be learned from video. A policy that cannot open cabinets also fails to learn placing objects inside one through video-only fine-tuning; video learning is capped by whether the policy can already execute the required atomic skills.

A few more caveats: the real-world evaluation covers 10 tasks at 5 trials each, so the 64% vs 56% margin carries limited statistical weight; future-probing on DROID still trails the teacher given real future frames, so the representations have not fully absorbed the predictive information; 11Hz is an H200 number; the bimanual section reports qualitative rollouts only, with no success rates; and LIBERO cannot separate methods when every baseline sits above 94%.

Terms

• WAM (World Action Model): a policy family that uses the predictive knowledge of pretrained video diffusion models for robot action
• VLA (Vision-Language-Action): policies adapting vision-language models to map observations and instructions to actions; the contrasting lineage
• REPA: a technique aligning a diffusion Transformer's intermediate layers with external representations to stabilize training; the distillation design follows it
• Flow matching: a generative objective regressing a vector field that transports noise to the target distribution; used for both the backbone and the action decoder
• Action chunk: a sequence of actions emitted at once (32 steps here), giving smoother control than per-step output

Source

• arXiv 2611.04527v2
• PDF
• Hugging Face

Related papers

• Frozen 5.46M LingBot-Touch lifts four policies by 13.9 to 24.8 points on 26 tasks
• BEHAVIOR-100 hits 0.6411 Q-score; filter recovery jumps from 14% to 94%
• Geometric symbols replace per-step vision; bridge-stack success goes from 24% to 56%
• Energy models orchestrate humanoid skills: 7x fewer policy switches, zero fine-tuning on a real G1
• Active harness search lifts LIBERO-Pro from 17.5% to 95% with frozen weights
• 119 Hz touch residuals cut finger MPJPE 30% between slow visual updates

All paper explainers