UrbanGround: From Local Perception to Spatial Agency in a Real-Scale City
Tianjie Ju, Zheng Wu, Yueqing Sun, Yuhan Cui, Bobo Li, Shengqiong Wu, Pengzhou Cheng, Haodong Zhao, Zongru Wu, Xinbei Ma, Doris Zhang, Kunling Li, Mong-Li Lee, Wynne Hsu, Hao Fei, Qi Gu, Gongshen Liu, Zhuosheng Zhang
cs.CV
2026-08-28
UrbanGround is a physically constrained Hong Kong replica for MLLM agents. GPT-5.5 hits 75% short nav; long-range success is near zero, closure safe progress at most 46.7%.
Multimodal models can already read a street photo: name the pharmacy, parse a sign, say what sits nearby. Urban action is the next step. After a corner, the sign is gone and the pose estimate still has to work. Game sandboxes are the wrong rules. Street-view hops have no continuous collision. Indoor nav is too small. The missing object is a city built from real geography, where each action changes the next frame of evidence.
UrbanGround loads the Hong Kong Lands Department 3D Visualisation Map and 3D Pedestrian Network into Unity: territory-wide oblique-photogrammetry tiles plus a georeferenced walk graph. The agent is a first-person character with continuous motion and collision against buildings and terrain; trajectories are logged in the same geographic frame. The pedestrian network is a reference, not a rail. Time of day, rain, and fog are controllable; Rocketbox pedestrians walk the sidewalks.
The observation is the current first-person RGB, optionally with a short frame buffer. Global tasks add a pan/zoom map that marks the agent and does not highlight a route. Each turn picks one action: move, sprint, look, jump, map select/pan/zoom/orbit, or terminate. Episodes cap at 100 steps; a move lasts at most two seconds.
Tasks form a five-level ladder with 810 human-verified instances. Level 1 is local QA (recognition, orientation, active search). Levels 2–4 stretch from named short routes to inferred goals and multi-stop plans. Level 5 injects a mid-route closure and moving pedestrians. Arrival is within 15 m of the target. Metrics include answer accuracy, navigation success, pedestrian-network adherence (PNA), safe progress under closure (SPR), and pedestrian collision rate (PCR). Ten models from GPT, Claude, Gemini, Doubao, GLM, and Kimi are scored with task-type-fixed prompts.
Local recognition is already usable. Heading and long-range control are not the same skill.
| Model | Visual Acc | Orientation Acc | Short nav | Long nav | Nav overall |
| Claude-Opus-5 | 91.3 | 58.3 | 48.8 | 2.5 | 17.9 |
| Gemini-3.6-Flash | 93.8 | 56.7 | 23.8 | 0.0 | 8.1 |
| GPT-5.5 | 82.5 | 40.0 | 75.0 | 0.0 | 20.8 |
| Kimi-K3 | 92.5 | 55.0 | 67.5 | 3.8 | 22.5 |
| Claude-Opus-4.6 | 85.0 | 46.7 | 75.0 | 1.3 | 22.9 |
On orientation, Gemini-3.1-Pro scores 23.3, near chance for four options. Active-exploration answers still reach 46–83, but agents often cut across the street and leave the walk network. GPT-5.5 and Claude-Opus-4.6 hit 75% on short routes; stretch the same point-to-point task a few blocks and success collapses. Instruction following, constrained paths, and multi-stop plans fail in the same pattern. Over half of long-nav episodes still end closer than they started, and multi-stop runs touch only 10%–20% of required stops: a local heading survives, the global route state does not.
Weather and time of day mainly hurt local QA; short-nav effects are uneven. Road-closure success is essentially 0 while PNA stays above 93%; Claude-Opus-5’s safe-progress rate peaks at 46.7%. Pedestrian collision rates sit at 76%–90%. Staying on the network is not the same as yielding.
Reading a street view does not imply finishing a walk through a real city. UrbanGround draws that break inside one geographic frame: atomic perception works, composition into sustained goal-directed behavior accumulates errors with no correction. Anyone building urban agents, embodied nav, or map tools should stop using short QA as a proxy for closed-loop travel. The sandbox is public, so memory, replanning, and pedestrian courtesy can be tested in the same city.
There is no standalone Limitations section. The mesh comes from oblique aerial imagery, so near-ground detail and interiors are missing. The walk graph does not constrain motion; PNA is a post-hoc check. A 15 m arrival radius is loose for city walking and may inflate short-nav. The 100-step cap is harsh on long routes, though human testers finished every instance under the same budget. Closures are announced by a system notice and a map mark, not discovered. The study scores MLLM agents with interaction history, not a specialist navigator, and never compares against a map-API shortest path. Hong Kong’s terrain and grade-separated walkways are unusual; a flatter city might look less grim.