Learning Heterogeneous Tactile Representation for General Dexterous Manipulation
Bing Han, Yichuan Peng, Haoyu Dong, Yuanmeng Hu, Yongyi Jia, Fangjing Wang, Kecheng Zheng, Yiming Zhang, Yanhong Zeng, Xiaoqing Zhang, Chen Song, Rundong Zou, Anqi Dong, Yuan Guan, Ka Leong Cheng, Xing Zhu, Yujun Shen, Nan Xue, Wei Ning
cs.RO
2026-11-10
A frozen 5.46M encoder, trained on ~5,000 hours of three sensor types, lifts four policies by 13.9 to 24.8 points on 26 real tasks; raw touch gives at most half the gain.
A dexterous hand has to know how hard a fingertip is pressing and whether the grasp is slipping. Vision does not see under the finger. Piezoresistive, magnetic, and visuotactile sensors all measure contact, and they do not share channels, layout, resolution, or sample rate. A model fit on one hand cannot read another, and the logs cannot share a training set. A fingertip on an edge is one physical event, whether a magnetometer, a resistive grid, or a camera behind gel records it. LingBot-Touch, from Robbyant, drops each sensor's own physics and keeps the geometry and timing of contact.
The corpus has 4,983 hours, 464,431 episodes, and about 1.33 billion frames across 24 task categories. Labels stop at the category. About two thirds comes from gloves people wear: 2,197 hours piezoresistive and 1,136 hours magnetic. The rest is teleoperation on Sharpa Wave hands, 1,650 hours of visuotactile fingertips. Pretraining draws the three sources with equal probability.
Each taxel carries a 3D coordinate and a finger id, placed in a unit-radius fingertip frame taken from that sensor's geometry. The address is five numbers, time, finger, and position, and it enters only through rotary embeddings, so the token count can change. A two-layer MLP for each sensing principle embeds two frames of one taxel, six scalars, as one token. A shared Haptic Transformer follows.
Tactile-JEPA predicts masked contact in latent space and does not reconstruct the raw trace. Most samples in a clip are zero, and a reconstruction loss fits that background. The context encoder sees visible tokens. An EMA target encoder sees the full clip. The predictor applies L1 on masked and visible positions, with weight 0.5 on the visible term. An 80% spatial mask is shared across the whole clip, so a hidden taxel cannot be copied from a neighboring frame. Over the first 300 of 200,000 updates the mask ratio rises from 40% to 80%. Contact usually hits only a few taxels. Masking 80% from the first step can hide all of them, and the gradients explode.
Training also resamples how densely the fingertip is read and shifts each taxel by at most half the gap to its neighbor, which blocks memorizing the three training layouts. A new sensor is a projection into the same fingertip frame plus the matching tokenizer. The encoder is not updated.
Both sizes use 12 layers. Tiny has 5.46M parameters, Small 21.76M. Real-robot runs use frozen Tiny, which adds 0.16%, 0.15%, and 0.13% to the parameter counts of T-Rex, π0.5, and LingBot-VLA.
Only the probe head is trained. The table is Small at about 5,000 hours. FTP-1 gets an extra input adapter per hand; the two numbers in that column are Sharpa, then XHand1. Sharpa's visuotactile sensors were in pretraining. XHand1's magnetic sensors were not. The force range is 0 to 30 N.
| Probe | Sharpa Wave | XHand1 | FTP-1 |
| Force MAE (N) | 0.81 | 1.32 | 2.31 / 3.72 |
| Contact center (mm) | 0.12 | 0.16 | 2.19 / 5.89 |
| Coverage MAE (pp) | 0.66 | 0.43 | 2.90 / 4.39 |
| Future force MAE (N) | 1.04 | 1.45 | 1.67 / 2.00 |
| Video recall, top-5 | 72.6% | 47.1% | 16.5% / 6.0% |
From 1,000 to 5,000 hours, Small cuts Sharpa force error from 1.68 N to 0.81 N and lifts recall from 42.5% to 72.6%. Tiny stops improving near 4,000 hours and loses retrieval recall at 5,000. Adding other sensors on top of a fixed amount of in-domain data helps seven of eight probe settings; magnetic force error falls from 4.34 N to 2.52 N. With the budget fixed at 1,000 hours, a single-sensor model wins on its own sensor in seven of eight settings. Visuotactile contact-center error is 0.34 mm for that model and 0.68 mm for the three-sensor mix.
At the same 5.46M parameters, a VAE and an MAE reconstruct the waveform. At 1,000 hours, contact-center error is 0.90 mm for latent prediction, 3.03 mm for the VAE, and 3.07 mm for the MAE. Recall is 32.1% against 4.4% and 5.6%. At 4,000 hours latent prediction reaches 0.32 mm and 49.3% recall. The VAE gets worse on four of five tasks.
The real-robot suite has 26 dual-arm tasks, 20 trials each, 200 demonstrations per task and 500 when both hands are required. Tiny here is pretrained for 4,000 hours and frozen. Means are over all 26 tasks.
| Policy | No touch | Raw touch | LingBot-Touch |
| Diffusion Policy | 21.6% | +7.0 points | 35.6% (+13.9) |
| π0.5 | 48.1% | +6.6 points | 69.9% (+21.8) |
| LingBot-VLA | 48.3% | +8.2 points | 73.1% (+24.8) |
| T-Rex | 45.4% | +11.2 points | 68.6% (+23.2) |
LingBot-Touch is best in 89 of 104 task-policy pairs and no worse than the original policy in 99. Raw touch hurts 22 pairs. On mahjong sorting, LingBot-VLA drops from 28.8% to 5.0%. On XHand1, the hand absent from pretraining, raw signals move the four policies by +4.6, -1.7, 0, and +2.0 points. The same signals through the encoder move them by +16.3, +24.5, +17.8, and +13.8. Zipper engagement goes from 11% to 55% for LingBot-VLA. SD-card release goes from 32% to 92% for T-Rex.
In simulation, PPO with three seeds on twelve tasks averages 69.0% with LingBot-Touch features, 45.6% with a CNN trained inside the task, 38.5% with the raw signal, 30.3% with binary contact, and 21.4% with no touch. Pen spinning reaches 97.1%, against 64.2% for the next best input. Across seeds, LingBot-Touch has a mean standard deviation of 3.9 points and a maximum of 9.2. The other inputs reach 55 on pen spinning.
On five Sharpa tasks, LingBot-VLA alone scores 55.9%. Pretraining the touch encoder from 1,000 to 5,000 hours moves that to 69.8%, 74.3%, 77.4%, 81.4%, and 82.2%. The FTP-1 encoder in the same policy scores 67.5%.
Gloves are how the hours get collected. The fingertip frame is how a new hand is attached. Frozen tokens, 40 per hand, condition Diffusion Policy or an existing VLA without a change to post-training. Given the same tactile hardware, raw readings deliver about half the gain and make a block of tasks worse.
Hours from another transducer buy force and short-horizon dynamics. At a fixed budget they cost some spatial precision. That is what one shared encoder charges.
The paper names three gaps. The experiments still use too few hands. Capacitive sensors were left out because the recordings on hand missed the quality bar. The canonical geometry and the corpus both cover fingertips, so contact on the palm and along the finger is barely learned.
The 26-task percentages are rubric scores. Entries such as 3.75% and 93.30% sit off the 5% grid, and 20 pass-or-fail trials cannot land on those values, so part of the 13.9 to 24.8 point gap is partial credit on substeps. Twenty of the 26 tasks run on Sharpa Wave, whose sensor was in pretraining. XHand1 has six. The policies use Tiny trained for 4,000 hours, not Small, which is the size that keeps improving out to 5,000 hours on the probes. Simulated touch is rendered from geometric penetration depth. Force error there is 3.264 N, against 0.81 N for the same style of probe on real Sharpa data, and the 69% average stays inside that simulated distribution. The public release is inference code, a small set of example recordings, and the Tiny and Small checkpoints. The 4,983-hour corpus is not in it.