LaLa-World: Scaling Latent World Models with Natural Language Actions
Junha Song, Geonmo Gu, Jeongeun Park, Heejeong Nam, Lucas Maes, Byeongho Heo, Dongyoon Han, Sangdoo Yun
cs.CV, cs.LG
2026-10-20
LaLa-World writes actions as text and trains a 0.8B LM for latent futures: 72.1% MPC success on five sim tasks, 7.5 points above JEPA-WMs, and 13.9× faster than five-step RAEv2.
JEPA-style latent world models skip pixels. A vision encoder turns each frame into features, and the predictor guesses only the next feature. Model predictive control (MPC) scores thousands to tens of thousands of action sequences against a goal image, which is affordable only if each imagined future is a vector, not a rendered frame.
The action interface stops that predictor from spanning domains. Sim and robot logs store numeric vectors whose size and meaning change by dataset: [2, 4] can be a planar move in one environment and a force in another. Images and web video usually have no such vectors, so DINO-WM, JEPA-WMs, and V-JEPA 2-AC train a separate predictor per domain and leave that footage out. Video generators already mix sources by conditioning on language. LaLa-World, led at NAVER AI Lab, keeps the latent objective and uses language as the shared condition, so one predictor trains on control trajectories together with broader images and video.
Numeric actions are written with fixed templates that keep magnitude and direction, for example moving a pusher 1.19 left and 0.68 up. Wall, PointMaze, Push-T, Meta-World, Rope, Granular, and the real-robot set DROID then share one predictor. RoboCasa is held out. Transitions without action labels use a description of the change: OpenVid captions, plus edit instructions from OmniEdit and ShareGPT. Template phrasing is narrow; these captions widen both wording and scenes.
The predictor is a language model, Qwen3.5-0.8B by default. The vision encoder stays frozen, DINOv2-S/14 or DINOv3-L/16, so the representation does not collapse. Text goes through Qwen's own tokenizer. Visual tokens pass a two-layer MLP into that embedding space, with 3D positional encodings indexed by time and image plane. The next feature is read from learned query tokens. The loss is squared error against the true next feature, one step only. Proprioception, the robot's joint angles and gripper state, is neither an input nor a target, because each domain defines it differently.
Earlier latent world models inject numbers through a modulation layer or by concatenating features, and cannot read text. A separate text encoder adds parameters. Using the language model as the predictor reads the sentence and writes the future in one network.
Planning follows DINO-WM and JEPA-WMs. The cross-entropy method (CEM) samples action sequences from a distribution, keeps the low-cost ones, and updates the distribution. Candidates are spoken with the training templates, rolled out in latent space, and scored by squared distance between the final feature and the encoded goal image. The simulator executes the leading numeric actions and replans when the benchmark asks for it. Baselines also omit proprioception, with the same horizon. A 12-layer decoder maps features to RGB with L1 and LPIPS, for viewing only.
LaLa-Policy fine-tunes this checkpoint. Action queries feed a GR00T-style diffusion head that emits an action chunk and the latent state N steps ahead. The encoder stays frozen. The action head starts from scratch. LIBERO executes 16 actions per step. RoboCasa predicts 32 and executes the first 16.
IRIS, DreamerV3, TD-MPC2, and LeWorldModel, trained without reward or task labels under the DINO-WM protocol, average 26.0%, 30.2%, 17.7%, and 28.1% on the five sim tasks. Those numbers are not their reward-trained settings. The fair latent baselines are the per-domain models.
| Benchmark | LaLa-World | Baseline |
| Five-sim average | 72.1% | JEPA-WMs 64.6%, DINO-WM 62.3% |
| Push-T | 81.2% | JEPA-WMs 68.8% |
| Wall | 95.8% | DINO-WM 90.6% |
| Meta-World Reach | 75.0% | JEPA-WMs 66.7% |
| PointMaze | 56.2% | JEPA-WMs 60.4% |
| Rope-group average | 50.5% | JEPA-WMs 44.8% |
| RoboCasa Place, zero-shot | 46.9% | V-JEPA 2-AC 34.4% |
| RoboCasa Reach, zero-shot | 40.6% | JEPA-WMs 37.5% |
| DROID action score | 40.2 | JEPA-WMs 43.0 |
The five sim tasks are Wall, PointMaze, Push-T, and Meta-World Reach and Reach-Wall. The second average covers Rope, Granular, both RoboCasa tasks, and DROID. LaLa-World leads both averages. PointMaze and DROID go the other way.
In Figure 4, adding image and video data moves the average from 59.3 on control data alone to 61.6. Paraphrasing the templates drops the control-only run from 59.3 to 52.5, and the full-data run from 61.6 to 61.0.
The RAEv2 comparison matches DINOv3-L/16, language conditions, and training data, with both predictors near 0.8B. RAEv2 is a flow-matching diffusion transformer started from public DiT weights, denoising once or five times per step. On 8 H100s, LaLa-World matches RAEv2's 12-hour planning score in 2 hours, then reaches 59.4% against RAEv2's best of 48.0%. At full depth on one H100 it plans 2.9× faster than one-step RAEv2 and 13.9× faster than five-step RAEv2.
On ImageNet, retrieving a transformed view from 256 candidates yields mean reciprocal rank 0.856 when the transform is described in language, against 0.024 for the source feature and 0.026 for an empty string.
CEM imagines 4,500 to 54,000 sequences per trajectory. Claude Opus 5 and GPT-6 Astra propose 8 to 96 language plans and receive only the goal distance. Claude reaches 100% on Wall with 90 rollouts, where CEM scores 95.8% with 9,000, and 53.1% on RoboCasa Reach with 80, where CEM scores 40.6% with 54,000. PointMaze falls to 31.2%, against CEM at 56.2%.
| Policy | Success | Time per trajectory |
| LIBERO, LaLa-World init | 97.7% | 0.86 s |
| LIBERO, Cosmos Policy | 97.3% | 4.45 s |
| LIBERO, CogVLA | 97.4% | not timed on this clock |
| LIBERO, random / Qwen3.5 init | 94.1% / 95.7% | same architecture |
| RoboCasa, 50 human demos | 57.3% | 2.12 s; Cosmos 11.71 s |
| RoboCasa, 300 MimicGen demos | 66.4% | Video Policy 66.0%, random init 65.1% |
LIBERO Spatial, Object, Goal, and Long are 98.6%, 100%, 97.2%, and 95.0%. CogVLA's Long score is 95.4%. With 50 demos, Qwen3.5 init scores 56.8%, random init 55.5%, and Cosmos-Predict2 init 56.0%. The Cosmos baseline used here drops future proprioception and value prediction, so both sides predict actions plus a future observation.
MPC is a search for the action sequence that lands nearer the goal image. With encoder, data, and language matched, latent prediction scores higher and plans 13.9× faster than five-step RAEv2. It also reaches that model's 12-hour score in 2 hours.
For policies, the pretrained predictor is an initialization. Under the same fine-tune it beats random weights and a bare Qwen3.5. On LIBERO the success gap versus Cosmos Policy and CogVLA is under a point, while a trajectory runs about 5.2× to 5.5× faster.
Image and video data add 2.3 points on average. The gain that shows up in the paraphrase test is the 6.8 points a control-only model loses when the template wording changes. The same language interface lets an LLM stand in for CEM: about one hundredth of the rollouts match CEM on Wall, and RoboCasa Reach scores higher with still fewer imaginations. PointMaze, which needs precise continuous control, does not follow.
The paper's own open items are concrete. The data and the 0.8B predictor are smaller than video generators such as Wan, and whether the image-video gain survives scale is untested. The encoder stays frozen, so features never adapt to future prediction. Supervision is next-step only, with no multi-step rollout loss. MPC horizons follow prior protocols and stop around six steps. Policies are evaluated only in the LIBERO and RoboCasa simulators. There is no value head, so best-of-N planning is out of scope.
The averages hide losses. PointMaze is 56.2% versus 60.4% for JEPA-WMs, and the DROID action score is 40.2 versus 43.0. Granular's standard error is about 14 points, so 58.3% versus 50.0% sits in the noise. The RAEv2 match covers the encoder, the language, this training run, and parameter count. The pretraining corpora differ: a public DiT on one side, Qwen3.5 on the other. The 6× figure is 2 hours versus 12 hours to catch RAEv2's score at the end of that budget. LaLa-World then continues to 59.4% while RAEv2's best is 48.0%, so the two runs are not stopped at the same point.
LIBERO at 97.7% versus CogVLA at 97.4% and Cosmos Policy at 97.3% is a saturated table. A gap under half a point is not evidence of a stronger policy. Qualitative actions only bin magnitude into slightly, unmarked, and far, while CEM still samples numbers. The paper shows successful episodes and does not report an aggregate success rate for that phrasing. Decoded frames look soft because the training loss never sees pixels.