T-WLA: Bridging Slow Vision and Fast Touch
Kaichen Zhou, Yuzhen Chen, Hannah Yao, Chonghe Jiang, Rao Fu, Xihang Yu, Wei Dai, Bashar Kabbarah, Haoyu Chen, Ishita Goluguri, Zhiyang Dou, Haonan Chen, Fangneng Zhan, Mengyu Wang, Wojciech Matusik, Antonio Torralba, Paul Pu Liang
cs.RO
2026-10-21
Vision at 39.7 Hz caches memory; touch at 119.1 Hz corrects only unexecuted actions. EgoTouch finger MPJPE drops 30% vs the best baseline; four robot tasks average 50% success.
In contact-rich manipulation, a sheet crumples or an egg slips in the few dozen milliseconds after the fingers touch it. A visual policy has to wait for the next frame and run the network again before the plan can change. Contact has already moved on. A pressure map can separate a firm pinch from a slip. It does not say which way the task wants the hand to go, or which steps of the current action chunk are still waiting to run.
Vision-language-action models (VLAs, which map images and language instructions straight to robot actions) and world-action models (WAMs, which tie actions to a prediction of the future state) usually share one clock. Image, representation, and action update together. Contact changes faster than that. T-WLA narrows the question to the gap between visual updates: how fast touch should revise a slow plan that has already been issued and has not finished.
T-WLA (Tactile World-Language-Action) uses one history-aware visual memory for two jobs. It emits a base action chunk, and it revises that chunk with touch before the chunk finishes.
The visual branch updates at about 39.7 Hz. Eight learned queries read the last K=8 stored visual outputs, eight tokens of width 960 each. A vision-language backbone mixes those tokens with the current image, the instruction, field of view, and proprioception. An action expert trained with conditional flow matching, a velocity field from noise toward the demonstrated action, turns the memory into a base chunk. Memory and chunk are cached together. The bank clears at the start of each episode and keeps only the latest K sets.
The tactile branch runs at about 119.1 Hz. Between visual updates it runs only the encoder and the editor. Each hand takes its latest eight 21 by 21 pressure maps. A CNN and a bidirectional GRU encode them, then subtract a fixed reference pressure window, so the features describe a change in contact. A small Transformer reads the cached visual memory, the base chunk, and the time already elapsed. FiLM, a scale-and-shift conditioned on another signal, lets touch modulate the action hidden state. The residual head edits only steps that are still unexecuted and whose hand has valid touch. Each correction is added to the original cached chunk, not stacked on the previous correction. A hand with no valid touch gets a zero edit.
Training has three stages. The visual branch learns from the action loss plus future-image prediction. The tactile encoder is contrastively pretrained on EgoTouch, on paired touch and hand-pose sequences. The visual branch is then frozen, and the editor trains on the cached memory and base chunks with a group-balanced Smooth L1 loss and a penalty on residual size. Future-image and future-pressure heads exist only in training. Inference drops them.
Until the next visual update, the base plan is assumed to stay a useful reference. If the hand never makes contact, the editor has no signal to follow.
Future hand pose is a macro-average over the five EgoTouch environments, plus Tachin's held-out test set. Baselines without a native touch interface get the same tactile stream, with the encoder and optimization budget aligned. T-Rex and T-WLA keep their own multi-rate interfaces.
| Method | EgoTouch finger MPJPE | Tachin finger MPJPE | Tactile rate |
| VITRA + tactile | 0.50 cm | 0.63 cm | 58.2 Hz |
| SmolVLA + tactile | 0.50 cm | 1.28 cm | 69.0 Hz |
| MemoryVLA + tactile | 1.34 cm | 1.87 cm | 15.0 Hz |
| T-Rex | 1.14 cm | 1.08 cm | 17.16 Hz |
| T-WLA | 0.35 cm | 0.46 cm | 119.1 Hz |
Wrist-relative finger-joint error drops from 0.50 cm to 0.35 cm on EgoTouch, about 30% under the strongest baseline, and from 0.63 cm to 0.46 cm on Tachin, about 27%. The wrist moves less. EgoTouch translation is 3.96 cm and rotation 6.10 degrees, against 4.08 cm and 6.27 degrees for the best baseline. The correction rate is 119.1 Hz, above SmolVLA at 69.0 Hz, the fastest baseline in the table.
The robot is a dual-arm Linker A7black, 7 degrees of freedom per arm, with Linker Hand L6 fingertip sensing and a RealSense D435i ego camera. Each task uses 30 teleoperated demonstrations. Before that adaptation, training is on human hands only. SmolVLA shares components with this visual branch.
| Controller | Egg transfer | Sheet pickup | Cup nesting | Bag dragging | Average |
| SmolVLA | 0% | 28% | 0% | 39% | 16.75% |
| SmolVLA + tactile | 0% | 30% | 1% | 37% | 17.0% |
| T-Rex | 4% | 3% | 22% | 63% | 23.0% |
| T-WLA w/o tactile | 18% | 72% | 17% | 82% | 47.25% |
| T-WLA | 21% | 78% | 17% | 84% | 50.0% |
Mean success is 50.0%, against 17.0% for SmolVLA plus touch. T-WLA without touch already reaches 47.25%. Touch adds 2.75 percentage points: sheet pickup 72% to 78%, bag dragging 82% to 84%, egg transfer 18% to 21%, cup nesting flat at 17%. A thin sheet needs enough force to stop a slip and not so much that the paper crumples. Nesting a cup also needs alignment and placement, which a grasp residual does not cover. On the same 30-demo budget, VITRA and MemoryVLA record no successful trials.
Removing touch raises finger error the most: EgoTouch from 0.350 cm to 0.430 cm (+22.9%), Tachin from 0.460 cm to 0.500 cm (+8.7%). Removing recurrent memory hurts the Tachin wrist, +21.6% in translation and +9.4% in rotation. Cutting memory from K=8 to 4 worsens the six metrics by 2.3% on average. Dropping both predictive objectives worsens them by 1.0%. T-WLA is lowest on five of the six, and ties the short-memory run on Tachin joint error at 0.460 cm.
With touch removed, the visual branch alone averages 0.65 cm translation, 3.64 degrees rotation, and 0.43 cm joint error on GigaHands. That is 25.3%, 15.9%, and 54.7% below SmolVLA, at 39.7 Hz against 23.0 Hz. Eight of nine dataset-level comparisons are best for this visual branch. The miss is TACO translation, 0.75 cm against SmolVLA at 0.72 cm.
What transfers is the split in clocks. The slow branch emits a history-aware action chunk. The fast branch edits unexecuted steps from cached memory, without rerunning the visual backbone, and the prediction heads stay out of the loop.
On the robot, 50.0% against 17.0% is mostly the visual branch. The no-touch variant is already at 47.25%. The roughly 30% drop in finger error is the figure that matches the editor's job. Removing the predictive heads costs about 1% on average, and those heads do not run at inference.
Thirty teleoperated episodes per task move an editor trained on human hands onto a Linker Hand. Mean success stops at 50%, and cup nesting at 17%. The interface is concrete enough to follow. The success rate is not high enough to deploy.
The base plan has to remain useful until the next visual update. When SmolVLA fails, the hand often closes beside the target, contact is weak, and adding touch moves success only from 16.75% to 17.0%. The editor adjusts a grasp after contact. Cup nesting stays at 17% with or without touch. The paper states that aggregate success does not show whether failures sit in the grasp, the alignment, or the placement.
Many ablation gaps are about 0.01 cm, and predictive supervision is about a 1% relative change, hard to separate from noise. The robot table does not report how many trials were run. The gap from 47.25% to 50.0% is 2.75 percentage points, with no evidence on whether it is stable.
Main metrics are averaged inside each sequence, then macro-averaged across scenes. They are a different number from a pool of every frame. An appendix contact gate uses a pressure threshold of 0.25, and the main text does not say whether Table 1 uses that subset. Of 192 action dimensions, 54 are padding, and joint error uses joints 1 through 20 in palm-local coordinates. The visual branch uses flow matching; the editor uses Smooth L1. Joint fine-tuning is only proposed in the appendix. The pressure map is 21 by 21, and the reference window comes from training-set statistics. Transfer to another tactile skin is untested.