EXIMO: VLM Guided Exploration of VLA Policies
Bhavya Sukhija, Oliver Groth, Mohit Shridhar, Tim Hertweck, Michael Bloesch, Markus Wulfmeier, Abbas Abdolmaleki, Martin Riedmiller
cs.AI
2026-08-20
EXIMO explores with Gemini, SFT on successful rollouts, then residual RL. On 22 simulated ALOHA tasks it adapts a 3B VLA without teleop and beats base-policy RL.
Most manipulation policies are large vision-language-action models cloned from teleoperation. Atomic skills such as pick and place already work inside the training distribution. A new instruction that needs reasoning or skill chaining, for example "put the fruit a monkey likes in the bowl," drops success. More teleop is expensive. Plain RL is sample-hungry on long horizons, and diffusion action heads are awkward to train with standard RL.
Sukhija et al. at Google DeepMind propose EXIMO for a narrow setting: a pretrained VLA that already has atomic skills, a natural-language goal, a success detector, and no extra human teleop. The job is to turn VLM common sense into skills the VLA can run.
Three stages: explore, imitate, optimize.
The base policy is the 3B Gemini Robotics On-Device model (GROD): a PaliGemma VLM backbone with a diffusion policy head, trained on Aloha teleop in real and sim, and commanded in natural language. The orchestrator is an off-the-shelf Gemini. At each step the VLM sees multi-camera images, the task text, and history, reasons in a think block, and emits a short instruction GROD can execute. Closed-loop rollouts go into a buffer only if the environment's ground-truth success detector marks them successful.
Imitation drops the VLM's subgoals as conditioning. The VLA is trained with ordinary behavior cloning on those successful trajectories, conditioned on the original task goal and predicting action chunks. Deployment then needs no VLM. Distillation also beats keeping the orchestrator at evaluation time.
The optimize stage does not RL-train the diffusion VLA directly. A residual policy outputs Δa, which is added to the VLA action. The residual is conditioned on state, goal, and the VLA action, and trained with off-policy MPO on a sparse success indicator. If SFT already yields non-trivial success, residual RL has something to explore from.
Evaluation is simulated Aloha, 22 manipulation tasks: two-item racking, semantic reasoning ("the item a monkey can eat"), left/right caddy slots, and multi-tool sequences. Atomic skills sit inside GROD's training distribution; composition, reasoning, and left/right spatial goals do not. Orchestration comparisons use 1,000 episodes. RL comparisons average 20 tasks across five seeds.
The paper plots success in figures and does not tabulate per-task percentages. The shape is consistent. VLM orchestration lifts success over bare GROD, especially on long-horizon chaining (PlateBowlOnRack) and reasoning variants (BananaInBowl-Reasoning), and successful episodes are shorter, so data collection is cheaper. Filtered SFT raises success further and beats live VLM orchestration at eval. Residual RL after SFT raises success again and shortens time-to-success. Giving bare GROD extra environment steps to match the explore-phase data still does not catch SFT+RL. On several tasks, SFT on VLM data alone already beats "base policy + more RL."
Two negative controls live in the appendix. Distilling orchestration into the residual, rather than into GROD, learns offline and then lags pure online residual RL, which the authors blame on distribution shift. Mixing VLM orchestration into a fraction of online episodes, then evaluating without the VLM, loses to residual RL that never saw the VLM: the buffer mixes "correct the orchestrated action" with "correct the bare VLA action." Free-form instructions and pick/place-only commands perform similarly on GROD, so this VLA does not need a constrained skill API.
This is a recipe for writing VLM knowledge into a VLA without new teleop, then taking the VLM off the robot at eval. For teams that already have a language-commandable policy like GROD, new-task data is the scarce resource, not another planner. Using the planner only to collect, distilling successful traces into the body, and polishing with residual RL is a better fit for real-time control than leaving the VLM in the loop.
The sample-efficiency story is concrete: the VLM shortens failed exploration, the success filter raises supervision quality, and the non-zero SFT success rate gives sparse-reward RL a foothold. Drop a stage and the figure advantage shrinks.
All numbers are from simulation; there is no real-robot result. Success uses a ground-truth detector. The authors list VLM success detection and automatic resets as future work; without those, a real closed loop still needs a person or a separate reward model. GROD already picks and places in distribution, so EXIMO is about composition and semantics, not learning manipulation from scratch. Residual RL is tied to a diffusion head; an autoregressive VLA would need a different optimizer. The 22 tasks are tabletop placement and tools-in-caddy, with no contact-rich or deformable objects. Figures are the primary evidence, so there are no citable per-task percentages.