Latent Action Research in Robotics: No Real Labels Needed
ylecun · x · 2026-07-20
Explores the cutting-edge application of latent actions in robotics: instead of directly predicting a robot's joint commands or controller inputs, it learns a compact latent encoding to explain the changes between adjacent video frames, completely eliminating the need for real action labels during training.
- Genie Model Validation: DeepMind's Genie model, proposed in 2024, serves as a clear proof of concept. It uses an inverse model to observe consecutive frames and infer discrete latent actions that explain the visual transitions. Even without ever seeing real action labels, the learned action space alone is sufficient to control the generated video frame by frame.
- Cross-Platform Transferability: This latent space is transferable and can be directly used to train imitation learning agents on videos without any specific robot annotations.
More from Embodied
- RealSense D585 Pro AI-Powered 3D Stereo Depth Camera Unboxed for Physical AI — chrismatthieu · 2026-10-26
- Physical world agent swarm demo shared on X — pduan · 2026-10-26
- Looking for a wearable mic with best battery life that feeds recordings to personal agents via MCP — altryne · 2026-10-26
- 'Lamp Swarm': a desk of lamp-shaped robots shows what physical AI work looks like — dee_hw · 2026-10-26
- Tencent's HiSTEVE 1.0: hierarchical VLA agent plays Minecraft with native keyboard and mouse, hits 85% SOTA — _akhaliq · 2026-10-26
- SJTU's VaM lets robots learn online from mistakes, 95%+ success over 4-hour autonomous runs — chris_j_paxton · 2026-10-26