Towards Predictive, Aligned, and Scalable Robot Learning
Peijun Tang, Shangjin Xie, Baifu Huang, Binyan Sun, Haotian Yang, Kuncheng Luo, Weiqi Jin, Shilin Fang, Jianan Wang
cs.RO, cs.AI
2026-07-13
Lumo-2 reframes action learning around latent world dynamics rather than reconstruction fidelity, and beats pi-0.5 and Fast-WAM on long-horizon and dexterous real-world tasks.
Vision-language-action (VLA) models that act as robot "brains" typically tokenize the camera image, the language instruction, and the arm action together into a transformer. The bottleneck is the action channel. The usual approach treats the continuous action either as a raw signal or discretizes it through a tokenizer trained with a reconstruction objective, so the goal becomes recovering the action signal as faithfully as possible. Reconstruction accuracy, however, is not control quality: representations optimized for low-level signal fidelity end up misaligned with the geometry control actually needs. Actions must also answer both vision (heavy with irrelevant variation) and language (abstract semantics), and naive joint training over these heterogeneous modalities is unstable, slow to converge, and weak at generalizing.
Lumo-2, from the Astribot team, targets exactly this misalignment between action representation and control, and makes multimodal alignment more controllable along the way.
The centerpiece is a latent world dynamics representation, denoted phi. It is distilled from two observation frames into the physically grounded future change that an action would induce, rather than predicting the next frame pixel by pixel. The central hypothesis: action generation quality is governed by the geometry of the latent space, and reconstruction loss biases the representation, so actions must be aligned progressively to world dynamics, vision, and language.
Alignment runs in three pre-training stages.
Stage 1 aligns actions with latent world dynamics. A dual-branch vector-quantized (VQ) architecture gives vision and action one autoencoding branch each, co-trained with a constraint that visual features contribute to action reconstruction (a-hat = DA(A, phi)). Training mixes Astribot S1 data, cross-embodiment datasets, and egocentric human video: 30,000 steps, batch 24, 64 H100 GPUs.
Stage 2 aligns actions with vision and language. A Qwen3.5-4B backbone with a semantic extraction module runs a set of multitask objectives (action reconstruction, behavior comprehension, vision-language-guided action generation, cross-modal prediction, cross-modal contrast) to make action tokens semantic. The vocabulary grows by 32 visual context tokens and 1024 semantic action tokens: 12,000 steps, 64 H100 GPUs.
Stage 3 co-trains on VLM, video, and robot data. The VLM corpus holds about 53M samples emphasizing localization, cognition, and planning. Inference uses Block-wise Autoregression (BAR), compressing 32-step token-by-token generation into 4-step block-wise parallel generation; end-to-end latency drops from 253.66ms to 93.53ms, a 2.71x speedup. 120,000 steps, 160 H100 GPUs.
First, does the world dynamics representation actually capture future-relevant physical information? Given only the initial observation, predict the task instruction: DINO features of the initial frame alone reach 43%; adding four future frames reaches 94%; and the initial frame plus latent world dynamics phi reaches 90%, close to the full-observation baseline. Phi does compress future-relevant semantics.
On VLM capability, against a Qwen3.5-4B baseline, Lumo-2 lifts several spatial benchmarks sharply:
| Benchmark | Qwen3.5-4B | Lumo-2 |
| VSIBench | 21.73% | 52.34% |
| Where2place | 55% | 74% |
| MMSI | 30.8 | 36.4 |
The real-robot suite is the main claim: 22 hard manipulation tasks across six categories, run on Astribot S1 against pi-0.5 and Fast-WAM. On temporal-reasoning tasks Lumo-2 leads clearly: collecting eggs from a conveyor belt scores 100% (92% for VLA-only fine-tuning); placing cubes on a rotating rack 81.67% (VLA 74.17%). Long-horizon tasks show the widest gap: making coffee and mixing a cocktail beat both baselines by a large margin, and Fast-WAM scores 0% on the cocktail task. On physical-understanding tasks (flipping an egg, driving nails), Fast-WAM fails completely on the egg, while Lumo-2 does it reliably. All seven dexterous tasks go to Lumo-2.
For anyone in embodied AI, this is an empirical case that action representation cannot be left to reconstruction. It takes a default assumption (optimize action tokenization for reconstruction) and argues against it, then engineers a staged alignment that pulls actions, world dynamics, vision, and language into one geometric structure, with the objective on control quality rather than signal fidelity. The near-3x speedup from BAR is also practical for real-robot control, where latency matters.
It is, plainly, a systems-and-engineering paper. The value is a complete pipeline that runs and performs strongly, not a single module you can drop into another stack.
There is no dedicated limitations section; the structure runs Related Work, Conclusion, Contributions. Read against the full text, several cautions stand out. Every real task is evaluated on Astribot's own S1 platform (a single embodiment), and the pi-0.5 and Fast-WAM baselines run in the authors' setup; with only 22 tasks and single-digit counts per category, the numbers should be read cautiously. Human-to-robot transfer (via VisionPro and egocentric video demonstrations) is shown qualitatively, without success-rate comparison. The phi probing accuracy (90% versus 94%) is a proxy metric, not task success. Stage 3 uses 160 H100 GPUs, a compute bar out of reach for most teams, which raises reproducibility questions.