Robbyant open-sources LingBot-Touch, giving robots fingertip touch sense with JEPA encoders

rohanpaul_ai · x · 2026-11-10

Robbyant released LingBot-Touch, an Apache 2.0 open-source model that turns fingertip sensor readings into signals robots' control software can use, addressing what cameras can't tell a robot: grip force and slip.

Three same-size encoders (5.46M params each) were trained on the same touch data. At 4,000 hours, the JEPA variant (predicting masked parts in latent space) hit 49.3% top-5 on tactile→video retrieval, while the VAE degraded to 3.1% — worse than at 1,000 hours.
The author attributes this to data sparsity and low signal-to-noise: fingertips only press a few spots, so most of each frame is near-zero noise that pixel reconstruction must copy unpredictably; JEPA can skip both. The paper cites the same reasons.
Training details: masking 80% of taxels from step one blew up gradients on contact-free clips, so the mask ramps from 40% to 80% over 300 steps; targets come from an EMA encoder at four depths, following V-JEPA 2.1.
Multi-source ablation: holding each sensor's own data fixed and adding only other modalities improves 7 of 8 sensor-task pairs, showing heterogeneous sensor data directly benefits individual sensors.

A chart shows simulated RL results: LingBot-Touch reaches 68.95% mean success across 12 tasks vs 21.42% with no tactile, 30.33% binary, 38.48% raw, and 45.59% with Tactile CNN.

Original post →

Robbyant open-sources LingBot-Touch, giving robots fingertip touch sense with JEPA encoders

More from Embodied