Sharpa's Roadmap: Turning Human Video into Tactile Data for Dexterous Hands
机器之心 · wechat · 2026-10-16
Machine Heart interviewed Sharpa's academic lead Zhang Kaifeng on why dexterous hands break the human-video data recipe: video captures motion but not force, and fine manipulation hinges on millimeters after contact.
Three research lines: (1) closing the embodiment gap — UVTA adds fingertip tactile sensing to an exoskeleton glove and jointly trains human demos with teleoperation data in a shared visual-tactile-action space, scaling from 0% to 84% success as human demos grow to 1,000; DexEMG replaces the exoskeleton with forearm EMG; Dex-X extracts hand-object trajectories from monocular video, completes contact forces in simulation, and distills to real robots (93% block grasping). (2) Decoupling motion from interaction — the CraftNet hierarchy (VLM at 1Hz, motion planning 10Hz, tactile control 100Hz), and MoDE-VLA where operators trigger RL-trained in-hand primitives via a foot pedal (89% vs 34% assisted success), lifting apple-peeling from 0% to 30%. (3) Cross-embodiment representations.
Google's Gemini 4 Argon beanbag demo also uses Sharpa hands; the hardware is now a platform for multiple frontier labs.
More from Embodied
- Opus-assisted open design: a $31k low-field head MRI sketched in hours — DreamFly_13 · 2026-10-18
- DIY robotics build: servo mechanism tested, wide yaw range but limited pitch — _Stocko_ · 2026-10-18
- Brain implants are restoring speech to people with ALS, Science reports — EricMTrautmann · 2026-10-18
- Meta wearables VP runs his AI agent across 3 phones and 23 pairs of glasses — nikola_mr64990 · 2026-10-18
- AMI-EV: spinning prism gives event cameras stable texture, Science Robotics 2024 — RexDouglass · 2026-10-18
- Neuralink patients control robotic arms, drones and games with thought alone — XFreeze · 2026-10-18