RxBrain: Joint Language-Vision Embodied Model

Haotian Liang · hf · 2026-07-18

RxBrain proposes an "embodied cognition" foundation model that processes language reasoning and visual imagination within the same planning sequence.

Method Highlights

Unlike VLMs that focus solely on scene understanding or world models that only predict future visual states, RxBrain assigns language to handle task decomposition, constraints, timing, and decision logic, while visual imagination manages world state prediction and joint subgoal planning.
The model adopts a unified multimodal Mixture-of-Transformers architecture, supporting the understanding and generation of language, images, and video.
The authors also built an automated training pipeline that slices embodied videos into planning steps and aligns text with visual state transitions to form joint supervision.

Evaluation & Results

They introduced RxBrain-Bench to evaluate whether a model can represent embodied plans as a joint structure of "text reasoning + visual imagination" rather than isolated understanding/generation capabilities.
Experiments show that RxBrain maintains robust embodied understanding and generation capabilities, outputting plans enriched with text reasoning, world state predictions, and joint subgoal planning.
When further extended to continuous robot action generation, it demonstrated promising real-world robotic performance without relying on large-scale action data pre-training.

More from Embodied