VAMOS: A Hierarchical Vision-Language-Action Model for Capability-Modulated and Steerable Navigation
Mateo Guaman Castro, Sidharth Rajagopal, Daniel Gorbatov, Matt Schmittle, Rohan Baijal, Octi Zhang, Rosario Scalise, Sidharth Talia, Emma Romig, Celso de Melo, Byron Boots, Abhishek Gupta
cs.RO, cs.AI, cs.LG
2025-10-24
VAMOS splits robot navigation into a VLM path planner plus a swappable sim-trained affordance scorer: 90% average real-world success, one planner for legged and wheeled robots.
Robot navigation has long had two camps. Classical modular stacks (state estimation, perception, planning, control) are reliable and interpretable, but every layer is hand-tuned for one robot and a bounded set of scenarios; porting to a new body means retuning the whole pipeline. End-to-end navigation foundation models bet on data instead, yet a single network must learn both where to go and what this specific body can physically do, which breaks once datasets mix robots with incompatible action spaces: stair-climbing quadruped demonstrations are useless to a wheeled rover.
The paper's baseline numbers are blunt about the second camp: NoMaD and NaVILA average 27% and 10% success across real-world indoor and outdoor courses, below a tuned modular stack at 53%. VAMOS asks how to pool heterogeneous multi-robot data into one generalist navigator while keeping each body executing only what it can.
The core move is decoupling: the high level handles semantics, the low level handles the body, and the interface is a 2D path in pixel space.
Language steering comes nearly free: 10% of the training data was chain-of-thought annotated by GPT-5-mini (four noisy variants plus a mirrored copy per ground-truth trajectory; describe the scene, rank the paths, keep the top three), co-trained with COCO-QA and Localized Narratives against forgetting. At test time, appending a preference after the goal coordinate, such as 'stay right' or 'take the ramp', bends the predicted paths.
Swapping robots keeps the VLM untouched; only the MLP and the low-level controller change.
Six real-world courses, five trials each: indoors a 20m hallway, a dim cluttered atrium and a 5m lab route occluded by a large obstacle; outdoors a 40m campus route with a 7-step staircase, a 20m forest course and a 15m down-ramp with snaring vines. Success rates in percent:
| Method | Hallway | Atrium | Lab | Campus | Forest | Down Ramp | Avg |
| Modular stack | 100 | 100 | 100 | 0 | 0 | 20 | 53 |
| ViPlanner | 100 | 100 | 0 | 100 | 100 | 0 | 67 |
| NoMaD | 60 | 0 | 40 | 0 | 0 | 60 | 27 |
| NaVILA | 20 | 0 | 40 | 0 | 0 | 0 | 10 |
| VAMOS | 100 | 80 | 100 | 80 | 100 | 80 | 90 |
The modular stack fails outdoors on perception; ViPlanner fails on the two courses that need long-horizon reasoning; the learned generalists mostly walk straight lines or bounce off walls in open scenes. VAMOS is the only method at 80% or above on all six.
On a cross-embodiment test (stairs and a ramp side by side, 10 runs each): the legged Spot stays at 100% with or without modulation; the wheeled Hound rises from 60% to 90%, its choices shifting from stairs (4/10) to the ramp (9/10). ViPlanner scores 0% on Hound in the same setup. Single robots benefit too: when the VLM predicts paths through obstacles in out-of-distribution scenes, modulation lifts success from 20% to 60%, the source of the abstract's 3x claim. Language steering was checked once with a VLM judge: 20/20 alignment. Offline, the pooled-data planner beats single-robot specialists on trajectory error across all four validation sets, statistically significant on most metrics (p<0.05).
Three transferable takeaways. Heterogeneous robot data can be pooled if the interface is action-space-agnostic, and 2D paths are a real-world-validated choice. Cross-embodiment transfer shrinks to training one cheap MLP in simulation, with no real-robot risk, far cheaper than fine-tuning a foundation model per robot. And the baseline numbers pour cold water on 'scaling solves navigation': generalist navigation models do not yet beat a well-tuned classical stack in complex real environments.
The honest framing: this is systems-level composition. VLMs, affordance learning and hierarchical control all existed; the contribution is the interface and thorough real-world validation, not a new algorithm.
Admitted in the paper: the training data is almost entirely static, and pedestrians usually move aside in the demonstrations, so the model handles nearby dynamic obstacles poorly and often overshoots or undershoots turns behind occlusions. The data mix was tuned heuristically; two cheap extra sources (CoTracker-pseudo-labeled video, iPhone ARKit logs) hurt performance and were dropped, with automatic mix selection left to future work. High-level inference runs at 1 Hz on an RTX 3080 laptop, 0.5 Hz on battery, too slow for dynamic scenes.
Reasons for skepticism beyond that: five trials per course means 20-point success-rate granularity, so an 80% versus 100% gap can be one sample. The cross-embodiment experiment used a VLM fine-tuned on 2K samples derived from 50 hand-labeled images (a paper footnote), so it is not pure zero-shot. The affordance scorer depends on an elevation map built from LiDAR and depth cameras, which pure-vision stacks cannot reuse as-is. In simulation, Spot's locomotion was a PPO-trained proxy policy rather than the deployed built-in controller, and proxy fidelity for more different bodies is untested.