$τ_0$-VLA: a Hierarchical Robot Foundation Model with World-Model-Guided Test-Time Computation
Xiaowei Cai, Yunuo Cai, Bingao Chen, Jingxiao Chen, Zhi Chen, Siyuan Feng, Tengyu Hou, Jingshun Huang, Han Jiang, Runkun Ju, Dong Li, Mingxiang Li, Shaowei Li, Xinchen Li, Yifan Li, Yi Liu, Zhongyuan Liu, Jianlan Luo, Junwen Miao, Ruiqi Ni, Buqing Nie, Mingjie Pan, Xinlin Ren, Jianheng Song, Jiaxu Wang, Peiqi Wang, Sen Wang, Xiaoyan Wang, Dafeng Wei, Dongming Wu, Pengwei Xie, Pu Yang, Hangjian Ye, Xiangyu Yue, Jinyu Zhang, Qinglin Zhang, Xueyong Zhao, Pengfei Zhou, Yue Zhou
cs.RO
2026-08-18
τ0-VLA searches subtasks with a world model, then a 40k-hour VLA executes. Hierarchy lifts long-horizon success from 27.5% to 45%; TTC lifts Book Organization from 6/10 to 9/10.
Long-horizon chores are sequences of consequential choices, not denser motor commands. If the robot salts a pan that is already seasoned, or hangs a bag that is already on the rack, cleaner tracking at the wrist will not undo it.
Most hierarchical VLAs emit the next subtask in one forward pass. They neither compare alternatives nor imagine the scene those alternatives would produce. A bad commit is noticed only after the world has changed. τ0-VLA treats "what to do next" as a compute-scalable inference problem. Subtasks sit between action-level search and language-only planning: sparse enough to justify extra compute, long enough that the resulting image actually looks different.
Two levels. A high-level policy keeps a correctable execution memory and, when unsure, spends test-time compute. A low-level policy then executes the chosen subtask.
The high-level stack has four modules. Proposal, value, and reflective heads are independently fine-tuned from a robot-pretrained Qwen3.5-9B; the world model starts from Step1X-Edit. At each step the proposal model updates memory from the current multi-view observation, the task instruction, carried memory, and the previous subtask, then emits a direct proposal. Routing reuses that same forward pass: mean token probability over the whole generation, plus mean logit margin inside the memory field. If either statistic falls below a task-specific threshold, search runs. Thresholds are calibrated per task on held-out data; routing itself does not call another model.
Search is a propose-predict-evaluate beam. Each retained branch samples N candidate subtasks. The world model, looking only at the head-camera frame, predicts the terminal image of that subtask. The value model scores the (instruction, candidate, predicted image) triple into five ordinal bins mapped to 0.05-0.95. The top B branches by cumulative score expand to depth D. Memories produced inside search stay branch-local. A reflective model then reads the observation-aligned real context plus the retained branch summaries and writes the subtask that goes to the controller. It may copy a candidate. It may not.
Memory is trained to repair itself. Demonstration memories are perturbed so they lag, run ahead, or contradict the visual state; the high-level policy must fix the record before proposing. Value and reflective training uses offline imagined rollouts from the proposal and world models, labeled against the ground-truth next step.
The low-level policy pairs a Qwen3.5-2B vision-language backbone with a Mixture-of-Transformers action expert and generates action chunks by conditional flow matching. State and action live in a shared 40-D space covering end effectors, joints, grippers, waist, and mobile base, with per-sample masks. One interface covers fixed-base, bimanual, and whole-body mobile control. Training uses 40,115 hours of heterogeneous real-robot data plus multimodal co-training, in three stages: knowledge-isolated co-training, full end-to-end, then a small task-specific fine-tune. Action chunks have horizon 30 and are integrated with 10 Euler steps.
Four long-horizon tasks on AGIBOT G1, 10 physical trials each. Direct-execution baselines feed the full task instruction to the same low-level policy. Successful Clean Room rollouts last about 8 minutes over 25 steps; the other three tasks run about 3 to 10 minutes.
| Method | Avg. success | Avg. progress |
| GR00T N1.7 | 2.5% | 45.29% |
| LingBot-VLA | 0% | 44.43% |
| π0.5 | 22.5% | 73.05% |
| τ0-VLA direct | 27.5% | 80.10% |
| Hierarchical Plan Once | 45.0% | 87.85% |
Hierarchy without search already moves Tomato and Egg Stir Fry from 0/10 to 4/10. The paper names the bottleneck: adding salt barely changes the image, so a memory-free policy repeats the step or skips it. Clean Room goes from 4/10 to 5/10 (progress 92.80% to 94.80%), with remaining misses concentrated on hanging the bag and later room transitions. Make Milk Tea is 5/10 for both variants; hierarchical progress is 91.92%, a bit below direct execution at 96.15%. Leftover failures sit at lid sealing and straw insertion.
Test-time computation adds another increment. Open-loop next-subtask accuracy on out-of-domain Book Organization is 50.0% for Plan Once, 57.5% for Best-of-N, and 74.0% for TTC. Closed-loop:
| Task | Plan Once | TTC |
| Make Milk Tea | 5/10, 91.92% | 7/10, 95.38% |
| Book Organization | 6/10, 66.67% | 9/10, 93.33% |
| Clean Room | 5/10, 94.80% | 7/10, 97.60% |
Book Organization has no fixed script, and that is where TTC helps most. Accuracy rises quickly at small extra budgets, then saturates.
Short cross-embodiment tasks turn the high-level policy off. On ARX AC One, Collect Laundry is 10/10 with 97% progress versus π0.5 at 9/10 / 88%. On a bimanual Franka makeup table, Makeup Puff is 10/10 / 95% against π0.5 at 7/10 / 73.75%.
This ports a familiar LLM trick into hierarchical VLAs: spend compute on the hard calls, pass through the easy ones. The world model is queried before committing to a subtask, not after a subtask has already been chosen. Against π0.5, memory plus hierarchy lifts average long-horizon success from 22.5% to 45% under the same low-level interface. Turning on TTC still helps on tasks with no canonical order.
The practical bar is high. The recipe wants subtask annotations, an image-edit-style world model, and a per-task low-level fine-tune. 40,115 hours plus a 9B high-level stack is not a consumer-GPU reproduction.
There is no standalone Limitations section. Ten trials per cell means a 2/10 swing is noisy. Stage-three fine-tuning and routing thresholds are task-specific; cross-task zero-shot routing is not tested.
The world model imagines a single head-camera frame. Multi-view geometry and proprioception are absent from that imagination. On Make Milk Tea, hierarchical Plan Once progress is worse than direct execution, so decomposition is not free. Contact-rich steps such as lids, straws, and egg cracking stay hard even when extra compute sits at the high level. Best-of-N already shares the world model and value head; TTC's extra pieces are multi-step expansion and reflection, and the paper does not split those two.