Learning Foresight without Explicit Trajectories for 3D Diffusion Policies
Zhongbo Zhang, Zaibin Zhang, Yifan Wang, Changbo Yan, Lijun Wang, Huchuan Lu
cs.RO, cs.CV
2026-09-18
A latent supervised by sparse future gripper states, without decoded waypoints, lifts DP3 to 62.8% on 50-task RoboTwin2.0, 71.93% on LIBERO-40, and 72% on five real tasks.
3D diffusion policies, DP3 included, are good at dense, geometrically grounded actions from the current point cloud. Multi-step manipulation also needs a sense of where the interaction is heading: which contact comes next, which configuration two arms should close toward. Standard policies leave that to the same action-learning objective.
Predicting waypoints, keyframes, or full trajectories asks the predictor for more than the controller may use. Once a predicted future becomes a tracking target, new observations struggle to correct the motion. The Dalian University of Technology group keeps less. During training, sparse future gripper states shape a latent. At inference the decoded points are thrown away and only the latent remains as soft conditioning.
The backbone is DP3: point clouds plus robot state, observation window N=3, horizon H=8, execute 6 steps then replan. A trend encoder maps that short history to a 256-d latent. An auxiliary head reconstructs K=4 future gripper targets at offsets {5,10,15,20} frames, 4-D per gripper (xyz plus opening), 32 scalars, mean squared error with λ=1, trained jointly with the diffusion action loss.
At inference the head is off. The latent enters the usual global-conditioning path. Extra gated FiLM is applied only at the UNet bottleneck, with zero-initialized projections so the branch starts as identity. The bet: future information may organize the chunk's structure, but should not rattle the down/up blocks that refine local motion. Parameters rise from 262.43M to 271.67M (+3.52%). On an RTX 4090, 10 DDIM steps, batch 1, latency moves from 50.28 ms to 50.90 ms.
On 50-task mixed RoboTwin2.0, matched DP3 with the same 64-d task embedding scores 56.1%; this method scores 62.8% and improves 38 of 50 tasks. The largest jumps sit on contact-rich, multi-stage items (rotateQRcode +37, placeshoe +26).
LIBERO-40 overall is 71.93% versus DP3's 37.08%, including LIBERO-Goal 8.73% to 77.53%. DexArt averages 59.25%, 7.25 points above DP3. On five SO101 real tasks (push cube, stack bowls, stack cubes, lift basket, handover bottle) the average is 72.0% against DP3 49.0% and SimpleDP3 43.0%.
| Setting | DP3 | This method |
| RoboTwin2.0, 50-task mix | 56.1% | 62.8% |
| LIBERO-40 overall | 37.08% | 71.93% |
| LIBERO-Goal | 8.73% | 77.53% |
| DexArt average | 52.00% | 59.25% |
| Real five-task average | 49.0% | 72.0% |
\The DexArt writeup states a 7.25-point gain over DP3; 52.00 is 59.25−7.25. The HTML table does not list DP3's DexArt total on its own.
LIBERO ablations pin the mechanism. A parameter-matched latent with the future loss off reaches 44.87%, only +7.79 over DP3. Turning future supervision on adds another 27.06 points. Conditioning on explicit future points scores 65.28%, 6.65 behind the latent. Bottleneck-only FiLM is 62.8% versus 55.8% for all-block FiLM and 52.9% for cross-attention: deeper injection of the future condition disturbs local denoising. On a six-task ACT check, 24.00% becomes 51.67%, so the trend is not tied to the diffusion UNet.
Foresight for a 3D diffusion policy does not require a planner, and it does not require tracking predicted waypoints at test time. Plus 3.5% parameters and 1.2% latency, the absolute gains on LIBERO-Goal and the real robot are not a rounding error. For anyone already on DP3, this is a conditioning-path patch. The useful idea is not "predict the future". It is "supervise a latent with the future, then keep only the latent".
The real robot is SO101, five tasks, 20 rollouts each. When the arm occludes the object, the trend estimate is as starved of geometry as the baseline; soft guidance does not invent missing points.
Failure analysis points at occlusion: the arm hides the object, the point cloud is incomplete, the trend estimate shifts, and precise contact or alignment barely improves. Offsets past the end of a demo clip to the last frame, so on short demos "future" is the terminal pose. LIBERO materializes stride-4 trajectories, RoboTwin measures original-frame offsets, so {5,10,15,20} is not the same physical lookahead across benches. The ACT study is six tasks and 10,000 steps; it does not travel to every imitation backbone. DP3's DexArt total is recovered from a stated delta. There is no head-to-head number against FLARE or ForeDiffusion in this 3D point-cloud setting; comparison to future-view prediction stays qualitative.