DexForesight: Grounding Dexterous Policies with Implicit Spatial Foresight
Zhongxi Chen, Yi Yang, Xinyi Chen, Mingzhe Ye, Yaping Li, Zhihong Liu, Congsheng Xu, Haonan Wang, Zhijie Deng
cs.RO
2026-11-08
World model on offline depth and hand kinematics is aligned into the policy, then dropped. DexJoCo success goes from 49.4% to 60.5%; three real tasks hit 58.3%.
A dexterous grasp sends fingers around the back of an object, where the cameras lose them. Naming the current frame is not enough. The policy also needs a working guess of how the next action will rearrange several fingers and the scene. That guess is spatial foresight: articulated hand motion, plus the scene change that comes with it.
World-action models tie future prediction to action generation. On a multi-finger hand this runs into two concrete problems. Depth covers only visible surfaces. Hidden fingers are localized from the URDF and joint angles, but requiring depth at runtime ties the controller to a reconstruction pipeline that must stay consistent while the hand and object move. Other models render a future image or spatial state and then pick an action. Texture and background barely change the grasp, yet every control step pays for generating them. DexForesight leaves depth and kinematics in training. The deployed policy takes multi-view RGB, language, and robot state.
An action-conditioned spatial world model is trained first, then frozen. Query alignment writes its predictive features into the policy. Deployment runs neither the world model nor a future decoder.
Hand geometry does not come from the image. Fixed points on URDF links are passed through forward kinematics with the logged joint angles, into the robot base frame. Each point stays glued to its link, so a finger outside the camera still has coordinates. Point-BERT encodes the current cloud and a short motion history. Those features join RGB, language, and state history in a backbone initialized from RynnBrain. The target is a per-point displacement, so links can move separately instead of collapsing into one palm translation.
Depth is a training signal only. Real demonstrations get offline stereo depth from StereoAnyVideo. Simulation renders depth from MuJoCo. OmniVGGT turns each RGB-depth pair into geometry tokens. Cross-attention injects them after backbone layers 2, 6, 10, 14, 18, 22, and 26. Layers 4, 8, 12, 16, 20, 24, and 28 mix the earlier injections with a per-token router. Residual scales start small so the new path does not overwrite the pretrained features. Future frames have no depth label. Depth only describes the current scene.
World queries read layers 12, 20, and 28. A 16-frame horizon is split into four bins, one timestamp each. Demonstrated controls, encoded relative to the current state, pass through an adapter and tell the queries which future to model. A hand-flow expert predicts 3D displacement of the anchors. A visual expert predicts how frozen DINOv3 features change from the current frame. Both experts predict the noise added to their targets and apply Smooth L1 to the cleaned values. The action condition is randomly cut to a prefix. Before actions are applied, a context loss asks the queries to reconstruct the current hand, robot state, scene, and instruction.
The real-robot corpus is about 370 hours and 60 tasks in five groups: arrangement, assembly and contact, rotation and articulation, deformable objects, and tool use. The depth stage uses about 100 hours and only the first three groups, which are also the groups later tested on the robot. Warmup adapts the backbone to the prediction targets. Dynamics pretraining then learns interaction on the full corpus. Neither stage uses depth. Depth fusion comes last: the geometry encoder stays frozen, the fusion layers train, and the backbone and prediction heads keep updating.
During alignment the world model is frozen and neither prediction head runs. Policy queries see only the policy's own RGB, language, and state, then project into the teacher feature space. One loss matches each token's direction to its world query. Another matches pairwise cosine similarity, so the set does not collapse into unrelated vectors. The action expert appends these tokens to its original context and keeps the behavior-cloning loss. The policy never observes the demonstrated controls. It absorbs control-dependent dynamics and cannot reproduce the teacher's conditional prediction point by point.
At inference the extra context is computed once per observation and reused while actions are sampled. The world model, depth encoder, hand-point encoder, and both future heads are removed.
Simulation covers 11 DexJoCo tasks, 100 trials each, on a Franka with a 16-DoF Allegro hand. DexForesight is built on GR00T N1.7. The world model is fine-tuned on DexJoCo demonstrations, and depth is rendered rather than estimated.
| Method | Single-arm | Bimanual | Overall |
| π0.5 | 53.0% | 13.6% | 35.1% |
| GR00T N1.7 | 61.5% | 34.8% | 49.4% |
| Fast-WAM | 60.2% | 39.8% | 50.9% |
| Cosmos3 | 59.5% | 37.4% | 49.5% |
| DexForesight | 76.3% | 41.6% | 60.5% |
Overall success is 60.5%, 9.6 points above Fast-WAM, the strongest baseline, and 11.1 above the GR00T N1.7 backbone. Single-arm rises from 61.5% to 76.3%, a gain of 14.8 points. Bimanual success is 41.6%, only 1.8 points above Fast-WAM at 39.8%. The headline average is mostly a single-arm result.
The real platform is a Sharpa North robot with 22-DoF Sharpa Wave hands. The same recipe sits on QwenVLA, pretrained on the same roughly 370-hour corpus. Close Box, Insert Battery, and Screw On Lid get 20 trials each, with held-out object positions. Screw also changes cup color. Every method is fine-tuned for 30 epochs.
| Method | Screw | Close | Insert | Overall |
| GR00T N1.7 | 40% | 35% | 70% | 48.3% |
| QwenVLA | 40% | 15% | 35% | 30.0% |
| WLA | 50% | 30% | 70% | 50.0% |
| Fast-WAM | 15% | 5% | 10% | 10.0% |
| Cosmos3 | 25% | 15% | 50% | 30.0% |
| DexForesight | 50% | 45% | 80% | 58.3% |
58.3% is 8.3 points above WLA, the best baseline on hardware, and 28.3 above QwenVLA. Across 60 trials that is 5 extra successes. Latency is 230.4 ms against 227.4 ms for QwenVLA, 3.0 ms more, reported as a 1.3% increase. Cosmos3 spends 930 ms generating a 16-frame future video. On the robot, WLA runs its world expert and Cosmos3 generates future video together with actions. Fast-WAM, like DexForesight, does not decode a future at inference. The split is how spatial dynamics enter the policy during training.
After the world model is removed, a linear probe can still read future hand motion from policy features alone. At 0.54 s, 3D error falls from 13.59 mm on QwenVLA to 10.01 mm, a 26.3% reduction, and the 95% confidence intervals do not overlap. At 0.13 s the errors are 3.92 mm and 4.12 mm, and the intervals overlap. Separation shows up from 0.27 s.
Attaching prediction heads directly to GR00T is weaker than query alignment. All figures below are overall success on the 11 simulation tasks.
| Variant | Overall |
| DexForesight | 60.5% |
| Hand-flow head on GR00T | 53.1% |
| DINO future-residual head | 45.5% |
| Both prediction heads | 49.5% |
| Depth fusion inside GR00T | 47.4% |
| Pre-action world queries | 52.7% |
| Shuffled action conditioning | 41.5% |
| No offline depth | 49.7% |
| Depth tokens in context | 42.0% |
| Future-action target | 48.4% |
The strongest direct-prediction control is the hand-flow head at 53.1%, 3.7 points above GR00T's 49.4%. DexForesight is another 7.4 points higher. Both heads together fall back to 49.5%. Queries with no action condition reach 52.7%. Shuffling the conditioning actions drops success to 41.5%, below the backbone. Replacing hand motion with a future-action target scores 48.4%. Removing offline depth costs 10.8 points in simulation, down to 49.7%. On the robot it falls from 58.3% to 43.3% (Screw 35%, Close 25%, Insert 70%), a 15.0-point drop. Replacing cross-layer fusion with depth tokens concatenated into the context leaves 42.0%.
The deployment overhead is small: no world model, no online depth, no future decoding, and 3.0 ms over QwenVLA. Scene geometry and occluded fingers move onto the offline data pipeline. A lab that already has demonstrations, a URDF, and stereo video adds a teacher at training time and no extra module on the robot.
The gain is uneven. Single-arm simulation improves 14.8 points over GR00T N1.7. Bimanual simulation improves 1.8 points over Fast-WAM. On the robot, Close Box goes from 15% to 45% and Insert Battery from 35% to 80% relative to QwenVLA, while Screw On Lid ties WLA at 50%. The method helps hand-object geometry under occlusion. It does not move bimanual coordination to another level.
On a matched backbone, query alignment beats direct future supervision. A prediction head makes the policy emit a future. Alignment only keeps dynamics that remain readable in the latent state, which is what is left once those heads are deleted.
The real-robot comparison is not a controlled experiment. The paper says WLA and Fast-WAM differ in architecture and training, so neither is a one-variable ablation. Fast-WAM at 10% looks under-trained. That is not evidence that explicit future prediction fails on hardware. QwenVLA itself scores 30.0%, below GR00T N1.7 at 48.3% on the same three tasks, so part of the 28.3-point jump is recovery from a weak base. The 8.3-point gap over WLA is 5 successes in 60 trials, and no confidence interval is reported for those rates.
Coverage is narrow. The three real tasks are one each from the categories that had depth labels. Deformable objects and tool use are absent. Twenty trials per task is a small sample. Simulation uses an Allegro hand, the robot uses a Wave hand, and the world model is fine-tuned again on DexJoCo. This is not one weight set moved from sim to real. Bimanual success at 41.6% is still far from reliable. The 0.13 s probe does not separate DexForesight from QwenVLA, so the extra foresight shows up later, and failures are not split into slips, misalignment, or early release.
Training is not cheap. The 100-hour depth pass depends on StereoAnyVideo, and the hand points depend on a URDF plus logged joints. Fusion is brittle: concatenating depth tokens yields 42.0% in simulation, worse than 49.7% with no depth at all. The paper offers this as a hypothesis, that concatenation disturbs pretrained multimodal features. That claim is not tested further.