ReferTrack: Referring Then Tracking for Embodied Visual Tracking
Hanjing Ye, Tianle Zeng, Jiazhao Zhang, Shaoan Wang, Zibo Zhang, Weisi Situ, Yuchen Zhou, Yonggen Ling, Hong Zhang
cs.RO
2026-07-22
ReferTrack splits embodied tracking into referring then tracking: it picks a target from indexed boxes as a supervised multiple-choice step, and a single-camera 4B model beats multi-camera baselines on EVT-Bench's distracted and ambiguity splits.
Embodied visual tracking (EVT): a mobile robot with only an onboard camera follows a target described in natural language. Recent vision-language-action (VLA) policies fold identification and trajectory planning into one model, which sounds tidy, but their chain-of-thought reasoning lives in abstract spatial latents that are hard to supervise and align poorly with actual detections in the image. When the model locks onto the wrong person, everything downstream fails. ReferTrack splits "recognize" and "track" into two steps and grounds the recognition step back in image-space bounding boxes.
The first step is referring. The model lines up detected pedestrians in the current frame into an indexed candidate catalog, ⟨ped₁⟩…⟨pedK⟩, plus ⟨NOEXIST⟩ for when the target is out of frame. A single Refer-CoT token picks one index, turning identification into a constrained multiple-choice problem solved by autoregressive next-token prediction, fully grounded in image space and fully supervisable.
The second step is tracking. The chosen box conditions the trajectory: the LLM uses the Refer-CoT token as a conditioning prefix to emit an action token, and an MLP head decodes it into M waypoints. To keep the target alive through viewpoint changes and occlusion, ReferTrack keeps a FIFO sliding window of historically chosen boxes, embeds them through an MLP, and adds them to the TVBI token (ETvBi = ETvI + Pbbox(b)). The current frame uses TVI only, with no current box injected, forcing the model to lock the target from historical geometric cues. Identification is sharpened by co-training on a custom Refer-QA set: 2–3 captioned pedestrians composited onto diverse backgrounds, with supervision to pick the queried index or answer NOEXIST.
ReferTrack uses a single forward camera, 4B parameters, and SFT only. On EVT-Bench:
| Split | Success rate | Tracking rate |
| Single-target (STT) | 89.4% | 92.5% |
| Distracted (DT) | 73.3% | 81.8% |
| Ambiguity (AT) | 74.1% | 85.7% |
The headline is matching or beating multi-camera baselines with one camera. On the distracted split, TrackVLA++ manages 66.5% and NavFoM 62.0%; on the ambiguity split, TrackVLA++ is 51.2% and CoMaTrack 57.5%. ReferTrack beats all of them. Real-world deployment runs on the Unitree Go2 quadruped and G1 humanoid at an average 10.6 Hz with cloud-GPU inference.
The hard part of embodied tracking is never "how to move," it is "tracking the right target." ReferTrack turns recognition into a supervisable multiple-choice step with a refer-then-track recipe that is directly reusable for embodied following and robot navigation, and it shows a single-camera 4B model can stand up to multi-camera setups on identification-heavy tasks.
The body lists no explicit limitations, but an oracle ablation is telling: given a perfect box (oracle), success on the distracted split jumps from 73.3% to 81.5%, so the bottleneck is identification, not motion planning, exactly the step that needs more work. The 10.6 Hz figure also depends on cloud GPUs; whether on-device deployment holds that rate is unaddressed. Evaluation is EVT-Bench only, so robustness under open conditions (lighting shifts, dense crowds) still needs more evidence.