KAIST's latent world model lifts GUI agent success rates up to 40.6% with 98.9% less latency

kaist-ai · hf · 2026-10-14

KAIST researchers propose a Latent World Action Model that rethinks pixel-space world models for GUI agents.

Problem: predicting and re-perceiving full future screens is computationally heavy, reconstructs appearance details irrelevant to action selection, and induces an identity shortcut where predictions mostly copy the current screen
Method: shift the prediction target to action-induced future information in latent space; a large multimodal model serves as the latent world model with shared representations feeding two heads — action selection and predictive transition modeling — plus an action-aware contrastive objective to counter the shortcut
Results: up to 31.9% and 40.6% relative success-rate gains over pixel-space world models in offline and online GUI settings, with inference latency cut by up to 98.9%

Code is open-sourced (sjyu001/latent2world).

More from coding & agent