HarnessVLN: Unifying Training-Free Embodied Navigation through an Agent Harness
Yang Chen, Lirong Che, Zhenyu Huang, Wenbo Fu, Chuang Wang, Xu Cao, Daqi Liu, Yuzhe Yang, Jian Su, Lan-Zhe Guo
cs.RO
2026-09-14
Training-free HarnessVLN validates MLLM navigation proposals against spatial evidence and geometry, reaching 60.8%/53.9%/76.0%/59.3% SR on R2R, RxR, HM3D-v2, and HM3D-OVON.
Trained navigation policies map pixels and language to actions, then need new trajectories when the task or the house changes. Training-free systems put a multimodal large language model (MLLM) in the planner seat: it reads the instruction, proposes a subgoal, and a task-specific stack grounds and executes it. Semantic reasoning got better. The surrounding pipeline still decides what evidence is kept, which proposals are legal, and how failures feed the next step.
A plan that sounds right can still be spatially unsupported. Seeing a target is not reaching it. Issuing an action is not finishing a subgoal. Stale views, unreachable goals, and collisions send later planning off the actual task state. HarnessVLN inserts a runtime that keeps checking planner proposals against spatial evidence, task progress, and execution outcomes.
Nanjing University, AgiBot, and Tsinghua wrap that runtime as an Agent Harness. The MLLM only proposes semantic operations. Before a tool runs, the Harness checks argument validity, evidence provenance and recency, geometric reachability, and consistency with the active subgoal and failure history. Each tool returns structured feedback (arrival, collision, unreachability, no progress), which updates internal state and the map.
Eight tools share one I/O contract:
Swap a perception or control backend without rewriting the planner protocol.
Two memories. Hierarchical event memory is task-centric. Working memory holds the current view, a bounded panorama history, the active hypothesis, and recent tool feedback, so it does not grow with trajectory length. Progress memory tags each subgoal pending, active, completed, or blocked, with at most one active at a time; completion needs a pose-aligned observation or a successful tool result, so the planner cannot advance the task by assertion. Reflection memory stores full traces of unreachable targets, inconsistent grounding, collisions, stalled progress, failed backtracks, and rejected stops, with location, time, and supporting views.
The Spatiotemporal Graph (ST Graph) is environment-centric. Place nodes merge spatially compatible observations; consecutive places get bidirectional NavigableTo edges for forward travel and backtracking. Entity nodes hold ObjectNav targets and instruction landmarks, with aliases, spatial hypotheses, confidence, and supporting views. ObservedFrom records from where and when an entity was seen; Contains records coarse place-entity links. Retrieval ranks places by semantic relevance, recency, and salience, minus a failure penalty, then expands the top-K by one hop into the MLLM context. The graph can grow; the prompt does not have to. Failures land as lightweight annotations on the graph; the full event stays in reflection memory.
The Navigation Executor is replaceable: Fast Marching Method (FMM) for planar motion, NavDP on stairs. Stopping is a request, not a privileged action. Fstop requires semantic identity, geometric validity, and task progress together. ObjectNav needs a visually supported target inside the stop radius; instruction following needs evidence for the last route segment, the cited landmark, and the grounded endpoint. A rejected stop is logged and triggers more observation, refinement, approach, or backtracking.
Perception uses GroundingDINO for open-vocabulary boxes and SAM for masks. Simulation cameras are 640x480 RGB-D at 79 degrees horizontal FOV. Main tables use GPT-5.5 for instruction following and GPT-5.6-luna for ObjectNav, matching AgenticNav's base model on R2R.
On VLN-CE R2R and RxR val-unseen in continuous MP3D:
| Method | R2R SR | R2R SPL | R2R OSR | RxR SR | RxR SPL |
| HSGM (prior zero-shot) | 47.9 | 32.8 | 58.7 | 41.8 | 25.1 |
| AgenticNav-GPT-5.5 | 55.0 | 48.4 | 65.0 | – | – |
| HarnessVLN-GPT-5.5 | 60.8 | 43.5 | 72.7 | 53.9 | 38.0 |
| ABot-N0 (trained) | 66.4 | 63.9 | 70.8 | 69.3 | 60.0 |
Against AgenticNav on the same GPT-5.5, R2R SR rises 5.8 points and OSR 7.7. Against HSGM, RxR SR rises 12.1 and SPL 12.9, with nDTW essentially tied (54.8 vs 54.9). R2R SPL is 4.9 points below AgenticNav: more episodes finish, the paths are less efficient. Trained leaders still sit higher; ABot-N0 reports 66.4% SR and 63.9% SPL on R2R.
ObjectNav on HM3D-v2 (1,000 episodes, fixed categories) and HM3D-OVON (3,000 episodes, open vocabulary):
| Method | Trained | v2 SR | v2 SPL | OVON SR | OVON SPL |
| DRIVE-Nav | no | 72.4 | 41.3 | 50.2 | 32.6 |
| MSGNav | no | 74.4 | 33.4 | 48.3 | 27.0 |
| HarnessVLN | no | 76.0 | 37.9 | 59.3 | 36.6 |
| ABot-N0 | yes | – | – | 54.0 | 30.5 |
On OVON the system tops every listed method, including trained ABot-N0 at 54.0%. On HM3D-v2, 76.0% SR edges MSGNav's 74.4%, while 37.9% SPL stays below DRIVE-Nav's 41.3%.
Cumulative ablations on locked 100-episode subsets: R2R SR goes 46.0 to 54.0 (memory), 60.0 (graph), 64.0 (stop validation); OVON goes 45.0 to 52.0, 53.0, 55.0. Stop validation trims the R2R OSR-SR gap from 15.0 to 13.0, but widens the OVON gap from 18.0 to 19.0 and drops SPL from 34.2% to 33.0%. Appendix model swaps on the same OVON subset: GPT-6-astra 63.0, Qwen3.8-flash 51.0; MSGNav with GPT-5.6-luna only 37.0.
Real-world tests run on a 1.74 m AgiBot A3U humanoid with Qwen-3.8-27B as planner and Fast FoundationStereo for depth. The paper shows corridor instruction following, an open-vocabulary fire-extinguisher search, and a signboard-then-vending-machine combo. No success-rate numbers.
The portable piece is the runtime protocol, not another planner prompt. Keep the MLLM as a semantic planner; pull spatial checks, failure writeback, and stop gating out of the model. Instruction following and ObjectNav share the same tool contract, so swapping FMM, NavDP, or a humanoid local planner does not require retraining a navigation policy.
For teams already running GPT-class models zero-shot, the 5.8-point R2R SR gain versus AgenticNav on the same model is the cleanest read of the Harness itself. On open-vocabulary object search the system even beats several trained baselines. Evidence checking is doing real work there.
This is not a free lunch. SPL does not rise with SR on R2R or HM3D-v2. Trained giants still win instruction following. The humanoid results are qualitative.
The authors say the current Harness depends on predefined orchestration and validation, and they flag self-evolving rules in open environments as future work.
Ablations use 100-episode subsets, so the 18-point and 10-point SR lifts do not map cleanly onto the full-val 60.8% / 59.3%. The two task families also use different base models. A shared protocol across tasks is shown; a shared model across all four tables is not.
Failure cases are specific. In R2R episode 839 the projected-target gate passes at 0.998 m, yet termination is 6.48 m from the goal: proximity to the chosen point is not the instruction endpoint. In HM3D-OVON 2469, fresh depth passes at 1.20 m while the benchmark geodesic is 2.15 m, so the episode fails. Sensor nearness and eval success are different rulers. Stop validation raises completion; it does not uniformly raise efficiency.
Three real-world examples, no statistics. Appendix model sensitivity is still the 100-episode subset.