Scaffolding Foundation Models into Physical-World Agents Pushes the Frontier of Long-Horizon Navigation
Zixing Lei, Gengze Zhou, Xiong-Hui Chen, Jiazhao Zhang, Yiyang Huang, Hang Yin, Haoqi Yuan, Qi Wu, Weixin Li, Siheng Chen
cs.AI, cs.RO
2026-08-31
NavMCP wires a VLM to a navigation model via intent, observation, and memory, hitting 76.7% on HM-EQA and 78.3% on a Unitree Go2, 14.9 points above a matched episodic interface.
Long-horizon physical tasks need two things at once: reasoning over a distant goal, and closed-loop motion that does not fall apart. Foundation models currently split that load. VLMs can fill in missing facts and revise a high-level plan, then become brittle when every step has to be grounded as a navigation action. Navigation foundation models (NFMs) turn a semantic goal into a closed-loop trajectory, then forget the task: one bounded episode, no memory of rooms already searched or hypotheses already killed.
Wrapping the NFM as an ordinary tool call hits what the paper calls the episodic interface gap. The instruction does not encode the evidence need, observations along the route are thrown away, and nothing accumulates across calls. NavMCP is scaffolding for that gap. Neither model is retrained.
The outer agent is a VLM, Qwen3.6-Plus in the main tables, Qwen3.5-397B-A17B in matched ablations. The executor is Qwen-RobotNav. Three channels structure the handoff.
Intent turns "what evidence is still missing" into one semantic navigation call: a mode, a natural-language sub-goal, a step budget, and optional constraints. navigatetoobject is object search; navigatebyinstruction is route or region language. The agent never emits low-level controls.
Observation turns a full rollout into source-grounded journey evidence. Keyframes are sampled along the path, then a VLM summarizer writes rooms, objects, uncertainty, and unexplored exits. Each mention is tied to a keyframe and a confidence label. "Not in this view" is not allowed to become "not in the world."
Memory keeps a compact EQA state across calls: interaction history, a source-grounded evidence ledger, and unresolved goals. Raw tool traces are compressed only after the ledger is updated. An answer must point back to the current view, a journey summary, a reviewed keyframe, or a notebook entry. Auxiliary tools cover panoramas, open-vocabulary detection, close-ups, and keyframe review.
In the cross-paper table, flagged as contextual only, NavMCP reaches 76.7% on HM-EQA versus 69.2% for FAST-EQA, 54.4% versus 50.5% on MT-HM3D, and an LLM Score of 79.27 versus 68.7 on EXPRESS-Bench. Normalized high-level step fractions are 0.15 and 0.19, against 0.52 to 0.65 for reporting baselines.
Matched conditions are the causal numbers. With Qwen3.5-397B-A17B, episodes, and budget fixed, Explore-EQA scores 57.6%, ToolEQA 60.8%, FAST-EQA 63.5%, and NavMCP plus Qwen-RobotNav-8B 74.0%. Replacing the three channels with an episodic interface drops accuracy to 59.1%, a 14.9-point hole. Channel cuts: terminal-only observation -5.9, no journey analysis -4.4, no cross-call state -4.6, single navigation mode -2.0.
The executor is not interchangeable either. Outer agent fixed, Random Walk 60.9%, Frontier 65.3%, StreamVLN 69.3%, Qwen-RobotNav-4B 73.3%, 8B 74.0%. No agent, initial view only: 38.2%.
On a Unitree Go2, 60 episodes (20 per horizon), NavMCP succeeds 78.3% of the time, versus 48.3% for frontier coverage and 38.3% for a reactive NFM. The gap versus the strongest baseline grows from 10 points in a single room to 25 across rooms and 45 beyond 20 m (60% versus 15% and 0%). Counting is the weak category at 65.0%; existence and identification sit above 80%.
For embodied-agent builders, the move is to wire a planner and a navigator through a persistent evidence interface rather than train one giant policy. Most of the gain is in what survives between calls, not in extra navigation parameters. On the robot, longer tasks widen the gap, which is a stronger claim than another simulator leaderboard bump.
The bill is also clear. The outer loop is still a large VLM, so inference is slow, and duplicate keyframes inflate counts.
The authors list two limits: a large VLM that slows inference, and cross-view duplication that hurts counting.
Cross-paper numbers in Table 1 are not causal. The normalized-step metric bills NFM methods extra per 3 m and keeps original accounting for non-NFM baselines, so efficiency comparisons lean in NavMCP's favor. Real-robot cells have 20 episodes and human success labels, so the intervals are wide. Main numbers use Qwen3.6-Plus; ablations use Qwen3.5-397B. Do not mix them. The scaffold is not tested on long-horizon manipulation outside EQA.