SALA: Semantically Aligned Latent Actions Enable Dual-Axis Scaling of Embodied Foundation Models
Jiachun Jin, Zetong Zhou, Haonan Wang, Zhijie Deng
cs.RO, cs.AI, cs.LG
2026-11-13
SALA aligns latent actions of unpaired human and robot clips by language similarity, letting human video directly supervise a VLA; 72.5% average success on four real-robot tasks, above LAM/WAM baselines.
VLA models generalize only as far as their interaction data, and teleoperation is slow and expensive to scale. Egocentric human video is the obvious supplement: recording it needs no robot. The obstacle is the embodiment gap. A person grasping a cup and a robot arm grasping a cup share almost nothing in pixels, viewpoint, or motion, so signals in human video do not reach robot control by default.
Latent action models (LAMs) are the popular fix: infer an action-like latent from consecutive frames without labels, then have the policy predict it on both human and robot video. But existing LAMs train that latent with reconstruction alone, which only constrains each clip against itself. The latents end up clustered by visual appearance, with same-scene actions bunched together and same-action clips scattered across scenes. Human and robot clips form separate clusters, so what the policy learns from human video means nothing for control. The alternative, world-action models (WAMs), predicts future pixels alongside actions, which is expensive to train and entangles scene appearance, so it collapses when scenes change.
SALA uses language to supply the cross-embodiment correspondence that reconstruction cannot.
Both choices are backed by ablations. Dropping the stop-gradient makes alignment bidirectional and costs 6.7 points on the clean setting (80.2 to 73.5): human latents have no action supervision, and pulling anchors toward them contaminates control. Swapping SoftVQ for hard VQ leaves the alignment loss flat, because graded similarities need latents that move smoothly while one-hot assignments change only on code switches.
The policy is π0-style: Qwen3-VL-4B plus a 0.6B action expert. Learnable latent queries predict the frozen tokenizer's token distributions under a KL loss shared by human and robot data; robot data add a flow-matching action loss. Deployment needs neither the teacher nor future frames. Training runs in three stages: pre-training on human video (frames only, no captions), mid-training on teleoperation, and per-task post-training on teleoperated plus human demonstrations.
Simulation uses RoboTwin 2.0, with 2,500 Aloha-AgileX trajectories (9.2 h) for the target robot and 27,000 ARX-X5 trajectories (68 h) standing in for human video, an order-of-magnitude gap that mirrors practice:
| Method | Type | Clean | Random | Avg |
| π0.5 | VLA | 70.7 | 46.6 | 58.7 |
| LingBot-VA | WAM | 75.8 | 38.4 | 57.1 |
| OpenWAM | WAM | 86.8 | 3.9 | 45.4 |
| RotVLA | LAM | 74.4 | 28.0 | 51.2 |
| SALA | LAM | 80.2 | 48.6 | 64.4 |
SALA starts from a general-purpose VLM yet beats every baseline initialized from a robot-pretrained checkpoint. OpenWAM reaches 86.8 on Clean but falls to 3.9 under domain randomization; SALA holds at 48.6. Consistent with this, alignment raises the median fraction of latent variance explained by action from 0.41 to 0.67 and cuts the scene share from 0.14 to 0.04.
The real robot is a 63-DoF wheeled humanoid (Sharpa North, two 22-DoF dexterous hands), pre-trained on 2,678 hours of human video plus 888 hours of teleoperation, 20 trials per task:
| Method | Gear | Pouring | Flower | Spoon | Avg |
| GR00T N1.7 | 9/20 | 10/20 | 6/20 | 8/20 | 41.25% |
| OpenWAM-α | 12/20 | 15/20 | 8/20 | 16/20 | 63.75% |
| Same arch, no latent actions | 8/20 | 5/20 | 4/20 | 5/20 | 27.50% |
| SALA | 15/20 | 16/20 | 11/20 | 16/20 | 72.50% |
Both scaling axes hold. Pre-training at 0/473/1,103/2,678 hours monotonically lowers validation L1 error on action prediction (16.8 to 16.4 ×10⁻³) with no extra robot data. With teleoperated demonstrations fixed, growing human demonstrations from 10% to 100% raises average success from 53.75% to 72.50%.
This adds a scaling axis for VLA training that is independent of teleoperation. Teleoperated data are hard to scale; task-specific human demos need only a person willing to perform the task, roughly 8-13 hours per task here for double-digit success gains. On a new task, recording more human demos looks like a better marginal investment than more teleoperation. Against the WAM route, skipping pixel prediction saves training and inference compute and holds up better when scenes change. The alignment recipe is platform-agnostic: any action-labeled robot dataset plus captioned unlabeled video can use it.
The authors flag two: the tokenizer needs atomic-action captions for its training clips (on the real robot, a 27,406-clip subset generated by a VLM and not separately audited), and scaling is verified only up to 2,678 hours. Weaker points on a close read: the simulation's 'human' videos are another robot arm, so the hardest part of the human-to-robot gap is covered only by the real-robot study, which has 4 tasks × 20 trials; the 72.5% vs 63.75% edge over OpenWAM-α is about 7 successful trials; the headline +45 points is against a same-architecture vanilla VLA at 27.5%, a weak baseline; pre-training L1 gains are on the order of 2% with no mapping from L1 to success shown; and the platform and dataset carry no stated open-source commitment, so reproduction is hard.