$N_0$-VTLA: Scaling Vision-Tactile-Language-Action Model with Latent Tactile Tokens
NeoteAI Team, Fudan TEAI Team
cs.RO
2026-07-27
First VTLA policy pretrained on touch at scale. Actions condition on a predicted future-contact latent, not the current frame; 63.8% vs 44.0% on 20 contact-rich tasks.
VLA (vision-language-action) models have made manipulation policies general, but they are effectively blind to touch, which leaves them weak on contact-rich tasks like insertion, folding, and screwing. Without sensing the moment contact is about to happen, the action cannot adjust in time.
Two existing ways to add touch are both wrong, the authors argue. Concatenating tactile tokens into the vision-language prefix fails because touch is sparse and mostly silent, and a prefix built for information-dense views gains little from it. Injecting the current tactile reading into the action pathway records contact already made, leaving the policy one step behind itself.
N0-VTLA takes a third path: it never feeds the current tactile reading. The action expert conditions on a prediction of future contact. The reasoning is direct. The moments where touch matters most are anticipatory, the millimeter-scale pre-load before a grasp closes, the catch of incipient slip. These live in the pre-contact blind spot, where a purely reactive signal arrives too late to shape the action that caused it.
The recipe has three parts. A tactile encoder subtracts a per-episode zero-contact baseline frame and encodes the difference with a frozen self-supervised visual encoder into 10 tokens; freezing is deliberate, to preserve the representation, let new sensors onboard via a light projection, and save memory. A latent tactile predictor reads the current tactile tokens plus vision-language context and outputs latent tokens z, trained to match the encoded tactile change over the next 50 steps under an InfoNCE contrastive loss plus an L1 reconstruction of a coarse future contact field. A three-stage rollout brings the pathway online: first ground the predictor alone (z retrieves its future-tactile target at 92.3% top-1 against 3.2% chance), then align the action expert with the prefix masked, then train end to end.
An offline-RL procedure, ALTER, improves the policy from deployment data. It trains a pairwise progress model from dense stage progress on clean demos and sparse preferences from object-drop and human-intervention events, freezes it, and appends a binary advantage label to the task prompt; at deployment the prompt always uses "positive".
Pretraining uses NeoData, the team's large-scale visuo-tactile corpus; they call this the first VTLA model pretrained on tactile data at scale. On the nine real-robot NeoReal tasks, N0-VTLA wins all nine, averaging 47.2% success against 29.4% for π0.5 (progress score 56.8 vs 42.3); ACT completes none. Socket plugging reaches 85% vs 60%, surviving daylight shifts and recovering from failed insertions.
In simulation, N0-VTLA averages 83.1% on eight UniVTAC tasks against 67.1% for the strongest external baseline InternVLA-A1, and 50.8% on twelve harder NeoSim tasks against 45.8% for π0.5. Dual-arm tasks are much harder: single-arm averages 73.8% but dual-arm halves to 39.4%, and several specialist baselines collapse to single digits (InternVLA-A1 at 1.0% dual-arm). Across all twenty tasks, 63.8% vs 44.0%. With ALTER on three long-horizon tasks, N0-VTLA reaches 95/80/75% against 90/75/60% for π0.5.
Touch is a recognized hard problem in manipulation, yet most VLAs are still purely visual. The contribution here is less "added touch" than a clean inductive bias: touch should anticipate, not react. For anyone in embodied AI, the predictive-latent-tactile recipe with staged integration is reusable, and this is the first time tactile pretraining has been done at scale.
The 39.4% dual-arm rate exposes the current ceiling: more complex contact and coordination remain unsolved. Reproducibility is a concern because NeoData and the tactile sensor are proprietary, so outsiders cannot easily benchmark against them, and external baselines run under the authors' protocol. The predict-over-react claim is a hypothesis validated on this setup; whether it dominates across all contact regimes is not systematically tested.