ALPS: Visual State Encoders for World Models
Yanqing Liu, Letian Zhang, Chengzhi Liu, Zijun Wang, Jieru Mei, Hongliang Fei, Yuyin Zhou, Cihang Xie
cs.CV
2026-10-22
ALPS trains a ViT-B from still images and text; one token set holds state, language, and pixels. Frozen, it hits 85.5% LIBERO success and matches 3.5× larger V-JEPA 2-L.
World models plan by predicting how a scene changes under actions, and many of them do that on tokens from a frozen visual encoder. If those tokens do not say where objects are or what state they are in, scarce interaction data is not enough to learn the dynamics. Two other interfaces usually live in separate models. Language alignment lets a written instruction point at the scene without a second vision backbone. Pixel reconstruction lets both the observed frame and an imagined future be rendered and checked.
Public encoders typically cover one or two of the three. DINOv2, DINOv3, and C-RADIOv3 transfer spatially, and several ship with no text tower. CLIP, SigLIP 2, and PE-Core align well to language and read out state much worse. MAE and VC-1 keep pixel detail and lag on the state probe. The mix is unstable: the same cross-view refinement that lifts SigLIP 2 and CLIP on the state probe drops ImageNet zero-shot accuracy and retrieval by 4 to 12 points.
ALPS is a ViT-B/16 trained from scratch on still images and text. No video and no robot data.
Stage 1 trains on DataComp-1B with four losses on the same patch tokens: contrastive alignment to the web caption and to one sentence of a synthetic caption, autoregressive captioning, pixel reconstruction with ℓ1, structural similarity, and gradient terms, and dense correspondence across crops and flips. Synthetic captions come from a frozen LLaVA-LLaMA-3 captioner whose vision backbone is CLIP ViT-L/14, so that model's visual knowledge enters only as text. The run sees 12.8 billion samples at 128 resolution, then 1.0 billion at 224. The image encoder, text tower, and linear projection are kept.
Stage 2 freezes a copy of those weights as the teacher. The teacher sees a clean crop covering 40% to 100% of the image. With equal probability the student sees a local crop (10% to 40%) or a global crop with photometric augmentation, and a predictor regresses the teacher's channel-standardized tokens at the matching locations. Geometry queries carry position and the side-length ratio of the two crops. Only teacher patches whose centers fall inside the student crop enter the loss. The objective holds local tokens steady across crop, scale, and appearance while anchoring them to the stage-1 features, a different job from filling in masked regions. The 5 million images are single frames from public editing collections. Edit pairs and instructions are unused. The predictor is discarded.
The language vector applies a linear map to each patch token and then averages. A linear map commutes with averaging, so zero-mean local updates cancel in the pool. SigLIP 2 uses nonlinear attention pooling and CLIP reads a class token, so the same update moves the global embedding.
Fifteen encoders stay frozen. A probe of about 2.5 million parameters reads the last 16 frames and estimates movable-object position, orientation, and on/in relations on LIBERO-Object and LIBERO-Spatial. Prediction settings also receive the next 1 or 3 seconds of robot actions. State error below is relative to ALPS stage 1, which is set to 1.
| Method | Metric | Result |
| ALPS (ViT-B) | LIBERO state-probe error, six-way mean | 0.829 |
| C-RADIOv3-B / DINOv3-B | same | 0.864 / 0.870 |
| V-JEPA 2-L | same, about 3.5× the parameters | 0.829 |
| SigLIP 2-B / PE-Core-B | same | 1.160 / 1.170 |
| ALPS | Flickr30k image-to-text R@1 / ImageNet zero-shot | 93.2 / 73.4 |
| OpenVision-B / PE-Core-B | best ViT-B on those two metrics | 94.5 / 78.4 |
| ALPS | ℓ1 of a decoder on frozen tokens | 0.120 |
| OpenVision 3-B / MAE-B | same | 0.104 / 0.109 |
| ALPS + native text tower | mean LIBERO policy success | 85.5% |
| SigLIP 2 + native text tower | same | 84.8% |
The aggregate ties V-JEPA 2-L, and the split goes the other way: 0.842 versus 0.741 on the current state, 0.813 versus 0.904 at a 3-second horizon. On objects that moved, ALPS averages 3.22 cm, inside the 3.04–3.25 cm band of V-JEPA 2, DINOv3, and C-RADIOv3. Against SigLIP 2 and PE-Core the aggregate is about 29% lower. Among probed intermediate blocks, the lowest test error is 0.985 at PE-Core block 8, still above 0.829. No compared encoder is better than ALPS on state, language, and pixels together.
All four retrieval scores sit within 2 points of the best mark. ImageNet zero-shot is 73.4, 5.0 points behind PE-Core, in line with training on long synthetic captions and testing on short class names. ℓ1 is 0.161 for C-RADIOv3 and 0.196 for DINOv3. MAE, VC-1, and OpenVision 3 reconstruct at 0.104 to 0.109 and carry 14% to 22% more state error. With the light decoder, C-RADIOv3 has a slightly better LPIPS, 0.359 versus 0.372. With a stronger decoder, ALPS ℓ1 is 0.107 against 0.141, about 24% lower.
Stage 2 cuts state-probe error by 17%, moves retrieval by at most 0.5 points, shifts ImageNet by about 1 point, and moves ℓ1 from 0.112 to 0.120. The same recipe drops SigLIP 2 by 7.5 points on Flickr30k image-to-text and 11.4 on COCO, and drops CLIP by 10.2, 10.9, and 11.8 on ImageNet, Flickr30k, and COCO. Cutting SigLIP 2's learning rate from 1e-5 to 3e-6 still reduces its state error by 17% (to 0.959) and still costs 2.3 and 2.9 retrieval points. Refinement on DataComp web images reaches 0.886, against 0.829 on the curated editing images.
Behavior cloning on frozen tokens, with 50 demonstrations per task, gives ALPS 83.4% mean success under BERT instructions, against 81.4% for SigLIP 2, 79.0% for DINOv3, and 78.2% for C-RADIOv3. Spatial rises from 76.8% to 86.5%. With the native text tower the main table shows 85.5% (Long 78.5%, Goal 87.4%, Spatial 82.6%, Object 93.7%), against 84.8% for SigLIP 2. Across three training seeds the native tower averages 85.2% and BERT averages 82.1%.
Planning trains one dynamics model per encoder. PushT success is 90%, tied with C-RADIOv3, but the final distance is 23.4 against 20.2. Wall success is 92%, tied with DINOv2, with the best distance at 3.21. On Granular, 10 goals, Chamfer distance falls from 0.324 to 0.176. Every encoder solves PointMaze. Rope Chamfer is 0.750 against 0.558 for C-RADIOv3, and Reacher success is 78.7% against 86.0% for DINOv3. Five-step prediction error falls 28%, 17%, 13%, and 11% on PushT, Reacher, Granular, and Rope. Rendered PushT rollouts have five-step LPIPS 0.020, against 0.018 for C-RADIOv3.
At one tenth of the stage-1 budget, dropping reconstruction raises ℓ1 from 0.113 to 0.194 or 0.200 and lowers state error by 8.5%. Dropping dense correspondence barely moves the global metrics and cuts cross-view patch matching from 37.4% to 31.4%.
One frozen token set can take an instruction, feed a dynamics model, and render a predicted frame. The text tower replaces BERT. The decoder is trained only on real frames and never enters planning. A DINO-WM-style stack, which freezes the encoder and trains only the dynamics, can swap in this backbone.
The margins are small. State error ties a video model with 3.5× the parameters, policy success leads SigLIP 2 by 0.7 points, and planning wins some tasks and loses others. The fit is a system that wants all three interfaces from one frozen encoder. A specialist reconstruction model or a video encoder is still stronger on its own axis.
The study stays at ViT-B/16 and in simulation. Larger scales and real robots are not tested. Balancing reconstruction, language, and state takes a two-stage curriculum, and a single stage is untested. There is no video supervision, so fine temporal actions are a weak spot, even though DAVIS video object segmentation rises from 0.456 to 0.533. Language-conditioned control and world-model planning are separate evaluations. The three interfaces are not yet inside one planner.
The 3.22 cm error on moving objects overlaps the 3.04–3.25 cm band of the strongest spatial models. Most objects never move: copying the true current position already yields 1.5 to 4.8 cm at 1 and 3 seconds, while moving objects sit at 12.8 to 31.6 cm, and every tested encoder is under 6 cm there. Stage 2 depends on object-centric editing images; DataComp web images only reach 0.886. The planner uses one recipe for every encoder, and a tied PushT success rate still comes with a worse final distance. Reconstruction is a light adapter into a frozen SDXL autoencoder, so it measures what a small probe can recover. V-JEPA 2 consumes 16 frames at 256 resolution, while the other models are 224-pixel image towers, so the protocol is not fully aligned.