World-to-Wrist: Task-Conditioned Future Wrist Modeling for Fine-Grained Robot Manipulation
Yuhao Pan, Haosong Peng, Zhengshen Zhang, Zhengyang Yan, Yalun Dai, Fushuo Huo, Chujie Wang, Tianyu Qi, Xiucheng Wang, Nan Cheng, Wenchao Xu
cs.RO, cs.CV
2026-08-06
W2-VLA forecasts future wrist latents conditioned on task context to give fine-grained manipulation foresight, hitting 98.5% on LIBERO and 33% OOD on bimanual plug insertion at 87 Hz.
Vision-language-action (VLA) models map camera images and language instructions to robot actions. Most multi-view VLAs treat the main-view and wrist-view cameras as two parallel visual inputs, encoded and fused equally. But the views play different roles: the main view gives global task context (scene layout, target objects, task progress), while the wrist view directly exposes the rapidly changing gripper and object interaction around the end effector. Fine-grained tasks like plug insertion need tight coordination between global understanding and local interaction dynamics, so treating the wrist view as just another input wastes it.
W2-VLA's angle: model not only the current wrist view but its near-future evolution, constrained by global task context.
W2-VLA is built on the StarVLA framework with a Qwen3-VL-4B backbone and a DiT flow-matching action head, about 4.97B parameters. The core is a task-conditioned latent interface connecting global context to wrist-future prediction.
First, the VLM contextualizes a set of 16 dedicated latent modeling tokens using the current multi-view observations and instruction; their final-layer hidden states form a fixed-length interface that encodes where the task is now and how the local interaction should move next.
Second, a wrist branch encodes wrist-history frames with a frozen V-JEPA 2.1 encoder and, conditioned on that interface, a bidirectional Transformer predicts future wrist latents, trained to match the encoded latents of real future wrist clips (available only at training, not inference). A Q-Former-style adapter compresses the dense predicted latents into 32 future-aware context tokens.
To shape the interface into a genuinely task-conditioned representation, the authors design W2-CoT structured annotations (synthesized by an offline VLM), each with three fields: Subtask (current stage and progress), Reasoning (robot-centric physical transitions such as approach-to-contact, grasp, transport, alignment, release), and Wrist (target proximity, fingertip contact, grasp stability; bimanual data uses a left and right format). These serve as auxiliary prediction targets during training only; they are not decoded at inference.
Finally, the VLM action context and the future-aware wrist context are concatenated and fed to the flow-matching action head to generate an action chunk.
On LIBERO, the average success rate across four suites is 98.5%, +1.3 over the strongest baseline (VLA-JEPA at 97.2%), with Spatial, Object, and Goal at 99.6, 99.8, and 99.2 and the long-horizon Long suite at 95.2. On the bimanual benchmark RoboTwin 2.0, the clean (Easy) setting reaches 60.71%, +7.79 over the strongest baseline UP-VLA, and the domain-randomized (Hard) setting 18.21%, +1.87 over π0.
In the real world on CoBoT Magic (built on Mobile ALOHA), three tasks stress different skills: table cleaning (long-horizon), occluded placement (global-to-local coordination), and bimanual plug insertion (fine-grained bimanual). Under standard conditions the average is 70%, +15.56 over VLA-JEPA and +28.89 over π0; under out-of-distribution perturbations (clutter, lighting, background) the average is 52.22%, +14.44 over VLA-JEPA. On the hardest task, bimanual plug insertion out of distribution, W2-VLA reaches 33.33% versus 10% for VLA-JEPA.
Efficiency is the design's selling point: the fixed latent interface generates a 16-step action chunk in 183 ms, an 87.43 Hz action rate, fast enough for real time, while variants that decode CoT explicitly at inference take over 1.5 seconds per chunk.
The ablation isolates each part. Removing the wrist predictor drops LIBERO from 98.5 to 97.5, with the largest fall on the Long suite (95.2 down to 93.6), so future-wrist prediction helps most for temporally extended tasks; removing W2-CoT supervision drops it to 98.0. On prediction targets, wrist-only beats predicting the main view or both: the main view is mostly static, so its future latents are insensitive to action-induced local changes.
Fine-grained manipulation (insertion, alignment, contact-sensitive tasks) is a recognized hard problem, and the wrist view is the observation closest to the action. W2-VLA casts "predicting the wrist future" as a latent prediction target constrained by global task context, giving the model foresight while anchoring it to task progress through structured annotations, all without slow explicit reasoning at inference. For robot-VLA builders it is a concrete route that balances visual foresight with real-time control.
The paper has no dedicated limitations section, but the results expose several. RoboTwin 2.0 Hard sits at 18.21% and bimanual plug insertion out of distribution at 33.33%, so absolute success on the most contact-sensitive tasks remains low and far from practical. Real-world evaluation covers only three tasks with 100 demonstrations each, a small scale that leaves generalization claims undertested. The small LIBERO gain (+1.3) partly reflects a near-saturated benchmark; the real lift shows up on the harder RoboTwin and real plug tasks. The method also depends on a frozen V-JEPA 2.1 encoder, which caps the quality of the wrist representation.