PanoVLN: Towards Effective Panoramic Vision-and-Language Navigation
Zhen Wang, Changpeng Wang, Zhe Liu, Zhangyang Qi, Yuxiang Lu, Zimo Zeng, Donglian Qi, Xi Chen
cs.CV, cs.RO
2026-09-28
Panorama drop-in barely helps. PanoVLN uses long-horizon actions and confidence-guided execution; a 4B RGB model hits 77.3%/78.0% SR on R2R-CE/RxR-CE Val-Unseen.
Most VLM navigators still see a forward camera. A side door or a corridor behind the agent only appears after a turn, so the robot explores extra and stops extra times to replan. An equirectangular panorama (ERP) unwraps a 360° by 180° view into one rectangle, so the next passage can already sit in the current image.
Replacing perspective frames with ERPs, while keeping the same trajectories, action labels, and execution length, barely moves success and can lower it. Short action targets only supervise what sits ahead. Public VLN routes rarely fork in view. VLM features also skew semantic: they name objects, not how hallways sit around them. PanoVLN changes those three pieces.
The policy is Qwen3.5-4B with a frozen PanoVGGT geometry encoder and RGB only. Actions are forward 25 cm, left or right 15°, and stop.
Each training state is labeled with the next H=18 expert actions. A panorama already shows the route after a turn; a short target wastes that. In the horizon sweep, perspective policies peak at H=4 and ERP policies at H=18. Farther steps are less reliable, so inference does not run the full 18. Confidence-guided execution (CGE) sums −log probability of the predicted actions and cuts the prefix once the sum exceeds budget B=1.2, with a floor of 4 steps. If the intersection is clear, the robot can yaw then walk; if probability mass spreads, it looks again.
Public traces rarely show several paths at once and name one. The work builds 98K routes in 800 HM3D scenes that must pass visible branches. Qwen3.8-27B writes instructions from first-person video plus eight-view compass images. A segment is kept only if motion order, the chosen entrance, and the stop location all match a clean replay. States sit on a stride-6 grid so neighboring 18-step targets overlap in 12 actions rather than 17, with extra samples at sustained-turn onsets and near stop.
The current panorama gets finer tokens; history is at most 10 frames taken from the last 100. Geometric features from the same RGB ERP are aligned by region, projected by an MLP, and added as a residual with scale α=0.2. Token count stays flat, and no depth image is used.
Evaluation uses R2R-CE and RxR-CE Val-Unseen in Habitat. The full model trains on R2R/RxR, DAgger, and the constructed set.
| Method | R2R SR | R2R SPL | RxR SR | RxR SPL |
| DualVLN | 64.3 | 58.5 | / | / |
| AwareVLN | 65.4 | 55.1 | / | / |
| CorrectNav† | 65.1 | 62.3 | 69.3 | 63.3 |
| PanoVLN† | 73.9 | 67.9 | 74.1 | 63.9 |
| PanoVLN | 77.3 | 70.6 | 78.0 | 65.9 |
R2R-CE success is 77.3, 11.9 points above AwareVLN at 65.4. RxR-CE is 78.0, 8.7 points above CorrectNav at 69.3. Navigation error is 2.83 m and 2.85 m. Inside the † constraint, PanoVLN† scores 73.9 vs CorrectNav 65.1 on R2R and 74.1 vs 69.3 on RxR.
With geometry off and 4,000 updates, ERP at H=18 scores 56.4 SR. Turn- and stop-aware sampling lifts that to 62.8. On one H=18 checkpoint, fixed prefixes of 1/6/12/18 actions score 64.6, 64.2, 61.4, 57.2 SR on R2R; CGE reaches 66.6. PanoVGGT moves that setting from 66.6 to 68.6 SR; UniK3D reaches 67.8, while DA2 and DAP drop. DAgger adds 5.3 and 7.0 SR points; the constructed set adds 3.4 and 3.9 more. Qwen3-VL-4B already hits 76.2 / 75.7; Qwen3.5-4B adds 1.1 / 2.3.
On a Unitree Go2 with an Insta360 X5 at 1.5 m, remote RTX 3090, about 12 GB, no scene finetuning. Twenty shared instruction-route pairs in Hallway, Office, and Campus. Under synchronous control PanoVLN finishes in 86.4 s on average at 25.7 cm/s, with 5.4 pauses and 7.4 policy calls. StreamVLN takes 117.8 s and 34.3 calls. JanusVLN, which queries every action, takes 412.9 s and 101.7 calls. Per-call inference is 1.08 s, slower than StreamVLN at 0.58 s. Fewer calls cut waiting to 13.6% of the trial.
A wider camera wastes itself unless the action target, the replan rhythm, and the branch density of the data move with it. The portable pieces are long-horizon supervision, confidence-truncated execution, dense labels at forks, and a frozen geometry residual that adds no tokens. A 4B RGB policy at 12 GB beats several older methods that used depth, odometry, or maps.
The action alphabet is still 25 cm / 15° primitives. The robot waits on a remote GPU. Groups already running NaVILA or StreamVLN can treat panorama plus CGE as the next baseline. Swapping the camera and leaving the horizon alone will likely reproduce the result that ERP barely helps.
There is no standalone Limitations section.
Real-world tests use 20 pairs per setting, synchronous stops, and 272 ms of network on top of inference. No onboard loop, no planning while moving. Campus is indoor-to-outdoor transfer; the text does not list per-scene success rates. CGE uses B=1.2 and Emin=4 with no full budget sweep. The 98K instructions are VLM-written then filtered, and the full model also stacks DAgger, which breaks a clean data match with some baselines. CorrectNav, the RxR number beaten by 8.7 points, sits in the † group. Geometry fuses into the current frame only. The horizon sweep ran 4,000 updates without geometry, so it is not the full model's H curve.