A scalar per patch from pre-trained ViTs enables fast moving navigation in the real world
Steeven Janny, Leonid Antsfeld, Christian Wolf
ECCV 2026 European Conferenc
cs.RO
2026-06-19
NaverLabs ran 966 real indoor runs over 24 km: a frozen ViT at one scalar per patch still hits ~95% success; vision trained from scratch only reaches 53.5% SR in sim.
End-to-end navigation almost always sits on a pretrained vision encoder, yet almost nobody has measured, at walking speed in a real building, which encoder to pick and how much of its feature map the policy actually needs. Swapping a backbone in simulation is cheap. Rankings often flip once the same checkpoint leaves the simulator.
NaverLabs Europe pins the question to static point-goal navigation. At the start of an episode the robot gets a polar goal that never updates as it moves. At deployment the only camera is a forward RGB stream: no lidar, no depth. Top speed is 0.7 m/s, control runs at 3 Hz. The physical platform is a Rookie robot in one office building: 966 episodes, 24 km in total. Simulation uses Habitat with a second-order motion model identified on the real robot, scored on 2,500 HM3D validation episodes.
The policy is the group's earlier stack with the depth branch removed. A frozen ViT emits patch tokens. A trainable projector compresses them. A two-layer GRU holds memory. PPO trains the rest. The action space is 28 discrete (linear, angular) pairs, linear speed in {0, 0.2, 0.4, 0.7} m/s. Reward is +2.5 on success, geodesic progress, a 0.01 per-step slack, and a 0.1 collision penalty that does not end the episode. Habitat's teleportation actions are replaced by identified inertial dynamics, because a 3 Hz loop cannot correct a command for 333 ms.
Encoders in the lineup: DinoV2, DinoV3, VC-1 (MAE on 4.3M images plus 4,000 hours of egocentric video), AM-RADIO distilled from CLIP, DinoV2 and SAM, and DUNE distilled from DinoV2, MASt3R and human mesh recovery. The from-scratch control is a half-width ResNet-18 trained entirely with the RL loss.
The actual intervention is the projector. Cross-attention pooling (CAP) is the usual Perceiver-style readout: K learned queries attend into ViT tokens and pass values downstream. Pure Attention Projection (PAP) drops the value map and treats the softmax attention itself as the feature. Each query yields one spatial heatmap; those maps, plus wheel-IMU odometry, are all the scene information the policy gets. Linear patchwise reduction (LPR) and PCA are the other bottlenecks.
Two training recipes. Either the projector and policy train from scratch for 500M steps, or a 360° lidar-like scan trains for 850M steps in sim and the RGB agent fine-tunes for 100M. Privileged sensors never appear at deployment.
Vision trained from scratch reaches 53.5% success in simulation. Frozen ViTs change the table immediately.
| Encoder | Sim SR | Real SR |
| ResNet from scratch | 53.5 | not run |
| DinoV2 | 63.4 | 64.3±5.8 |
| VC-1 | 86.3 | 95.2±6.7 |
| DUNE | 90.7 | 95.3±3.3 |
| AM-RADIO | 85.2 | 100±0.0 |
| DinoV3 + CAP | 0.0 | not run |
Real eval is 3 runs of 14 episodes. Multi-teacher models AM-RADIO and DUNE are the most reliable on the robot. Generic DinoV2 lags. DinoV3 with CAP is 0.0 across three seeds.
Fine-tuning from the lidar policy lifts most encoders to 87–90% SR in sim and 93–100% on the robot. DUNE's real SCT moves from 26.8 to 30.1. AM-RADIO is the exception: real SR falls from 100 to 92.9 after the same recipe. The lidar teacher itself stays at 100 SR and 36.5 SCT in the real building. RGB has not caught it.
PAP at K=1, one scalar per patch, still gets DinoV2 to 92.9 real SR; K=2 reaches 100. CAP at K=1 hits 97.6 real SR, but then the heatmaps are not the whole feature. PCA, which discards layout, drops DinoV2 to 51.5 / 29.7 sim SR and AM-RADIO to essentially 0. DinoV3 only works with spatial bottlenecks: PAP at K=8 reaches 95.3 real SR, LPR is usable, CAP stays at 0 even with ViT-S and extra queries.
The heatmaps grow into navigable space, obstacles, and goal-agnostic "where could I go next" affordances. Matched viewpoints in sim and on the robot light up the same regions.
Three usable takeaways for anyone shipping visual nav. Do not learn perception from the RL signal alone. A frozen encoder plus inertial simulation is enough for sim and real tables to sit in the same ballpark. Generic self-supervision is not automatically the robotics default: VC-1's ego data mix and the multi-teacher models beat DinoV2/V3. The policy can live on one number per patch. The deliberately dumber PAP readout is the only one that can decode DinoV3 when CAP cannot.
This is not a new backbone. It is a real-robot factorial of encoder, projector, and privileged pretraining. Incremental as a method paper, concrete as a measurement.
Real testing is one office, 14 routes per config, and a 1 m success radius against 0.2 m in sim. Wheel-IMU odometry still gives pose relative to the start, so this is not vision-only localization. The appendix is candid: the hidden state resets inside 2 m of the goal and the goal is fed in again; a 2D lidar safety layer stops collisions, hidden from the policy but modeled in training. Any sim SR below 60% is never sent to the robot, so the DinoV3+CAP collapse has no real number.
The task is narrow: static point-goal, no language, no image goal, no dynamic obstacles as the main setting. Calling the sim2real gap negligible rests on similar metric magnitudes, while success radius, the safety bumper, and the reset protocol are not matched. The DinoV3 story (CAP too complex for a weak reward) is a post-hoc reading. There is no spectral probe that pins it down.