ALPS: a unified visual state encoder for world models hits 85.5% on LIBERO
yuyinzhou_cs · x · 2026-10-23
- The ALPS paper introduces a pretrained visual state encoder for world models arguing representations should unify physical-state information, native language alignment, and pixel reconstruction — capabilities existing encoders specialize in only one or two of.
- Method: a ViT-B/16 pretrained from scratch on still images and text with language, reconstruction, and spatial-correspondence objectives, then refined via cross-view self-distillation.
- Results: lowest aggregate LIBERO state-probe error among ViT-B models across a 15-encoder benchmark (matching the 3.5x larger V-JEPA 2-L); frozen ALPS tokens reach the highest mean policy success on LIBERO (85.5%) with the native text tower; matches or beats leading encoders on world-model planning (PushT, Wall, Granular); predicted future tokens decode directly into pixels.
- The team released Stage 1 and final checkpoints, 8 LIBERO policy checkpoints, and JAX/PyTorch training and evaluation code.
Related event: UCSC, Google and UCSB Release ALPS, a Unified Vision Encoder(3 posts)→
More from Research
- Sophont's Holon-S1 predicts genes and biomarkers from routine pathology slides — iScienceLuvr · 2026-10-23
- Glasgow study: raised eyebrows and slow blinks make speakers look confident — Limor_Raviv · 2026-10-23
- IFM open-sources RL360, the agentic RL infra behind K2 Horizon's training — rupspace · 2026-10-23
- Wayfarer discovers RL options from a single experience stream, SOTA on hard Atari games — MarlosCMachado · 2026-10-23
- PhysVidBench accepted at EMNLP 2026: do video gen models grasp physical commonsense? — aykuterdemml · 2026-10-23
- EMNLP 2026 paper teaches multimodal LLMs to get cartoons via incongruity-resolution supervision — aykuterdemml · 2026-10-23