Shared 4D features track and forecast TCP motion, 7.60 mm ATE on DROID

TCP4D: A Unified Tool-Center-Point Tracking and Generation Model for Robot Manipulation

Haosong Peng, Xiangyang Liu, Zhengshen Zhang, Yunfeng Fan, Tianyu Qi, Xin Xia, Yufeng Zhan, Wenchao Xu

cs.RO, cs.CV

2026-11-18

TCP4D recovers metric TCP poses from monocular RGB and generates future trajectories from the same 4D features, with 7.60 mm ATE on DROID and 69.19% mean success on RoboTwin.

What problem this solves

Manipulation videos are easy to collect, and synchronized action labels are often missing. Vision-language-action models and world action models train on action supervision. RGB by itself does not produce a command a robot can run.

The label worth recovering is the tool-center-point, or TCP: an SE(3) pose, position plus orientation, in task space. Any arm that can reach that pose executes it with its own inverse kinematics. Joint angles do not have to match.

The TCP is a virtual origin. It often sits in free space between the fingers, so the pixel can land on the background. Feed-forward 4D point trackers return surface positions, not tool orientation. Recovering a full pose from those tracks needs robot-specific priors and several rigid-fit stages, and error stacks up along that chain.

Method

TCP4D runs tracking and future-action generation on one 4D scene representation. Input is a monocular RGB clip plus one 2D point per arm on the first frame.

The encoder follows 4RC. A DINOv2 transformer alternates attention inside a frame and across frames. A Dual-DPT head, the dense prediction decoder, reads out metric depth and camera rays. Ray direction and origin store intrinsics and extrinsics relative to the first frame.

The TCP branch does not track a dense cloud on the gripper. It samples a 3×3 patch around the query, adds an arm embedding, and reads four backbone levels. Those tokens cross-attend to the scene and self-attend among themselves. The head regresses translation, a continuous 6D rotation, gripper opening, and a confidence. The rotation is projected to SO(3).

One center cannot determine orientation. Supervision expands each TCP into seven virtual points: the origin and six points 3 cm out along the local axes. Matching that rigid set ties down translation and rotation together, with a geodesic rotation term, a frame-to-frame velocity term, and gripper cross-entropy.

At generation time the tracked positions are projected back into the image with the predicted intrinsics. Local features are packed with the current joints, the gripper state, and noisy future actions, then passed through a diffusion transformer that denoises the action. Conditions are the current scene, a frozen DINOv2 encoding of the wrist image, and a frozen T5-base encoding of the instruction. Training is flow matching. Inference takes four Euler steps. The policy sees 4 history frames and predicts 32 future steps in the current camera frame. A two-arm action is 20-dimensional at each step.

Stage one trains a tracker on 90% of 33,966 Franka episodes from DROID, and a separate tracker on RoboTwin, both starting from 4RC weights. Stage two fine-tunes on RoboTwin only and trains the generator, using 50 clean demonstrations per task. The tracker is about 1.52 billion parameters. The full model, frozen encoders included, is about 2.27 billion.

Cross-embodiment augmentation does not need logged source-robot states. Poses move into a shared world frame, TCP conventions are aligned, and the target arm is solved with inverse kinematics. SAM 3 removes the source robot, video inpainting fills the hole, and the target robot is rendered in. Base orientation stays fixed. Base position may be searched, and every frame has to solve. RoVi-Aug and OXE-AugE assume recorded source end-effector poses. Here those poses come from RGB.

Results

Tracking is scored on 100 held-out DROID episodes and on 100 unseen RoboLab tasks. No baseline tracks a TCP directly. The comparison fits a rigid pose to point tracks, with ground-truth points on the gripper and a scale taken from the robot model. TCP4D receives one 2D query on the first frame, then runs 9-frame windows and reprojects its own prediction as the next query. SE(3) metrics are computed after a similarity Sim(3) alignment. MPE is the position error without that alignment.

SettingMethodAPDATEMPESuccess
DROIDTCP4D74.78%7.60 mm11.85 mm100%
DROIDBest baseline14.04% (CoTracker3)67.65 mm (MoSE3)91.76 mm (CoTracker3)89.64% (MoSE3)
RoboLabTCP4D43.71%16.07 mm28.42 mm100%
RoboLabBest baseline9.90% (VDPM)not statednot stated96.33% (VDPM)

On DROID, relative rotation is 1.03°, relative translation is 3.40 mm, and gripper accuracy is 95.84% at a 0.5 threshold. Depth within 3 m, after per-frame median alignment, has AbsRel 0.0330 and MAE 31.27 mm. VGGT-Ω, DA3, and MapAnything sit between AbsRel 0.199 and 0.228 in the same table. Moved to unseen RoboLab, the DROID-trained tracker has AbsRel 0.1718, behind CoTracker3 at 0.1142, and relative rotation 6.80°, behind MoSE3 at 0.99°. Gripper accuracy falls to 81.92%.

After DROID pretraining, three UR5 episodes from Berkeley AUTOLAB cut ATE from MoSE3's 44.56 mm to 12.22 mm and raise APD from 14.39% to 48.27%. Ten episodes reach 8.85 mm ATE, 64.28% APD, and 98.48% gripper accuracy.

The augmentation study trains Diffusion Policy, not the TCP4D generator. Four tasks contribute 50 Aloha-AgileX demonstrations each, retargeted to 199 ARX-X5 clips and 198 UR5-WSG clips. Mixing source and synthetic data lifts Aloha mean success from 53.5% to 65.5%. On ARX-X5, joint training moves mean success from 30.5% with synthetic data alone to 72.0%. Place Burger Fries goes from 12% to 78%, and Click Bell from 14% to 70%. UR5-WSG moves only from 32.0% to 39.0%. The abstract's 41.5% is those 41.5 points on ARX-X5.

As a policy, TCP4D averages 69.19% on 48 RoboTwin tasks, 100 trials each, under clean-to-clean. ACT is at 29.25%, RDT at 33.92%, DP3 at 54.27%, π0 at 45.21%, and EventVLA at 65.71%. TCP4D uses a third-person view. The other entries are marked as head-camera methods. Under clean-to-random, TCP4D falls to 10.94%, below Xiaomi Robotics-0 at 17.85% and π0 at 16.15%. The generator was trained on clean demonstrations only.

On a real CoBoT Magic robot, placing pens and stacking bowls use 60 demonstrations and 30 trials each, with cuRobo for inverse kinematics. Success is 83.33% and 73.33%, mean 78.33%, against 63.33% for DP3 and 31.67% for ACT. Mean stage progress is 3.50 out of 4. Inference does not use measured depth.

Why it matters

TCP is one task-space interface for reading motion out of video and for sending motion to a robot. RGB with no logged joints can become a demonstration, or the 32-step future the policy is asked to produce.

Ten demonstrations move the tracker onto a UR5, so it is not locked to the Franka footage in DROID. Mixed with source data, the synthetic clips change Diffusion Policy on ARX-X5 from 30.5% to 72.0%. UR5-WSG stops at 39.0%. A new arm does not inherit that gain by itself.

The 69.19% number is a clean-scene result. Randomization cuts it to 10.94%, and the real-robot evidence is two large-displacement tasks. The practical use is labeling TCP from video and building cross-embodiment demonstrations when source states were never logged. The paper does not show a general policy that should replace a VLA.

Limitations

The authors state the clean-to-random drop. The generator trains on clean demonstrations only, and broader multi-embodiment data is left for later.

Several comparisons do not line up one to one. Tracking baselines receive ground-truth gripper points and a metric scale from the robot model. Failed windows stay in the success rate and drop out of the error averages, so baseline ATE ignores failures. TCP4D's success rate is 100%, so its error covers every window. ATE is an RMSE after Sim(3) alignment. MPE is the unaligned position error. On RoboTwin the cameras differ: third-person for TCP4D, head cameras for the others. The 41.5-point gain belongs to Diffusion Policy trained on mixed data. The generator is not retrained on the synthetic set. The UR5 few-shot result is tracking only. Augmentation still needs a target kinematic model and a successful inverse-kinematics solve on every frame. Inpainting artifacts and query drift across windows are not measured. The real-robot section is one platform, two tasks, 30 trials each.

Terms

Source

Related papers

All paper explainers