Dream What I Can: Sparse Visual Planning for Long-Horizon Robot Manipulation
Chongkai Gao, Wenlong Huang, Haonan Chen, Lin Shao, Yilun Du, Jiajun Wu, Ruohan Zhang, Li Fei-Fei
cs.RO
2026-10-18
DreamCan mines executable sparse subgoals from action-free video, finetunes Wan2.2, and hits 91.2% on LIBERO-Long with a 0.1B policy.
Video models often draw a future that looks plausible and still cannot be run on a robot. Language does not pin down contact. On Push-T, an instruction to push the T-block into place never fixes which face to touch. Dense video locks every frame, so errors accumulate, and once execution drifts the remaining frames fight the real trajectory. The model also has to roll out a long clip autoregressively. Uniform sampling keeps uninformative motion, backtracking from failed attempts, and states the low-level policy cannot reach.
DreamCan keeps a short list of subgoal images. Each image has to advance the task and stay attainable.
The planner and the controller exchange images only. Gradients do not cross. From the current observation and the instruction, the planner emits K subgoal frames. The controller is a 0.1B goal-conditioned flow-matching Transformer that turns noise into an action chunk, conditioned on several upcoming subgoals together. It replans from the latest view, at 1.03 seconds per planning step on an RTX 5090.
Subgoals are labeled automatically on action-free video. Reachability is a goal-conditioned offline RL value: reward -1 before the goal and 0 at arrival, discount 0.95, expectile 0.75, so the fit prefers higher-return transitions in the data. Normalized R is the expected discount raised to the hitting time, between 0 and 1. Frozen RADIO features encode frames. A temporal Transformer pools history and fuses goal and language. Task utility U is cumulative progress. Most tasks finetune RoboMeter with LoRA, updating only a low-rank residual. Language Table, which cares about explicit layout, uses a VLM progress score plus SAM masks, rescaled to 0 and 1.
Dynamic programming picks K indices. Difficult jumps pay the negative log of reachability, a spacing penalty keeps the gaps from collapsing, and terminal utility is subtracted so the endpoint is pulled toward completion. Consecutive picks are at least δ apart. Given the scores, filling the table is O(KL^2). The final frame is appended. The box-folding example uses 12 subgoals.
The generator is Wan2.2-TI2V-5B. LoRA tunes the diffusion Transformer. The VAE and the text encoder stay frozen. Because the causal 3D VAE compresses time by 4, encoding sparse frames directly would merge neighbors into one latent, so each subgoal is repeated four times. Training pairs mid-trajectory observations with the annotated subgoals that follow, which is what closed-loop replanning requires. When language variety is low, the instruction is dropped with probability 0.5. Prefix training uses the current frame alone with probability 0.3 and otherwise freezes a clean head, with loss only on the tail.
The low-level network stays fixed. Only the plan changes. LIBERO-Long uses 379 demonstrations and Push-T the official 206. The first four rows share that policy.
| Method | LIBERO-Long | Push-T |
| End-to-end policy | 79.2% | 58.0% |
| Language planning | 75.6% | 62.0% |
| Dense video | 84.0% | 70.0% |
| Uniform subgoals | 82.4% | 72.0% |
| π0 (3.3B) | 85.2% | 76.0% |
| SEER (0.3B) | 87.7% | 68.0% |
| π0.5 (3.3B) | 92.4% | 86.0% |
| Lingbot-VA (5.3B) | 98.5% | 89.0% |
| DreamCan (0.1B) | 91.2% | 86.0% |
DreamCan and the first four rows are means over 50 episodes. The 0.1B policy, without large-scale pretraining, beats π0 and SEER, and ties π0.5 on Push-T at 86.0%. On LIBERO-Long it remains under π0.5 at 92.4% and Lingbot-VA at 98.5%. VLA-OS at 0.9B scores 76.0% and 72.0%.
On make-a-line, language planning scores 56%, dense video 71%, uniform subgoals 69%, and DreamCan 86%. Against VLP, on 100 test videos per task, the three Language Table tasks move from 56%, 78%, and 15% to 86%, 100%, and 52%. Training sets contain 1,424, 1,303, and 1,378 videos, with 15 subgoals each. Canonical shape was added by this paper. FVD on make-a-line falls from 233 for dense video and 258 for uniform frames to 154.
Ablating the selector on LIBERO-Long: spacing only 81.8%, close to uniform subgoals at 82.4%; without utility 74.2%; without reachability 78.0%; without spacing 83.8%; all three terms 91.2%. After a reset onto the current subgoal, a segment counts as success if objects arrive within 5 cm of the next one. Selected subgoals sit near that ceiling.
The reachability score follows the shortest path in the data. In a suboptimal Square demonstration the nut is grasped at frame 101, dropped at 107, and grasped again at 208. Anchored at frame 101, R drops and then returns to 0.80 at the regrasp, with the state within 1.4 cm of the anchor. Time decay alone gives 0.004.
The policy trains on 100 skilled demonstrations and then freezes. The planner takes up to 200 extra suboptimal videos. DreamCan rises as that pool grows. Dense video rises slowly. Uniform subgoals fluctuate because they cannot skip regressions. On 100 make-a-line test episodes, anchored search, which commits a good prefix and regenerates the tail, beats best-of-N over the whole sequence. Dense video under the same search still ends below DreamCan with no search.
On a bimanual YAM robot, 20 episodes per task, cardboard folding with 100 teleoperated demos scores 5% end to end, 0% dense, 10% uniform, and 40% for DreamCan. Packing covers 205 feasible combinations, 110 seen with 5 demos each and 95 held out: 0%, 5%, 20%, 65%. Lining up six cubes, from 90 demos and 3 target shapes: 10%, 0%, 25%, 55%.
Images are the only interface, so either module can be replaced on its own. With useful subgoals, a 0.1B policy that was not pretrained at large scale ties 3.3B π0.5 on Push-T and sits 1.2 points behind it on LIBERO-Long. Suboptimal video without actions updates the planner alone. Prefix conditioning turns test-time search and language steering into a concrete procedure: sample several plans from one observation, then rank them with a progress function written by a VLM.
The compute moved, from full dense rollouts to a 5B video model that emits a few frames, plus occasional resampling. The low-level policy itself does not add a new contact skill.
Real-robot rates stay modest: 40% for folding and 55% for making a line, 20 trials each, and no variance is reported. Canonical-shape plans succeed on 52% of tests. Reachability assumes the controller is broadly compatible with the recorded behavior. There is no check against a weaker or stronger low-level policy. The LIBERO-Long numbers for π0.5 and Lingbot-VA are cited from earlier papers. Only Push-T was rerun under a shared protocol.
The paper's prose says uniform subgoals beat dense video. In the table that holds on Push-T only. On LIBERO-Long dense video leads 84.0% to 82.4%, and on make-a-line 71% to 69%.
Search reuses the training progress function. On Language Table that function is scripted detection. Gains when progress cannot be scripted are untested. Language steering also needs a VLM to write the score. Scaling with suboptimal data is shown on Square alone, with the policy frozen and the planner retrained per subset. Sensitivity to the DP weights and to K is not fully reported. The 0.1B figure counts the low-level policy. The planner is still a 5B Wan2.2.