360CityArena: A Realistic Virtual Urban Navigation Benchmark for Embodied Agents
Kenta Watanabe, Atsuyuki Miyai, Mizuki Takenawa, Kiyoharu Aizawa, Toshihiko Yamasaki
cs.CV, cs.AI, cs.LG
2026-08-10
A 360°-video reconstruction of Akihabara tests embodied agents on 175 urban tasks; the best model scores 17.1% versus 77.3% for humans, and map navigation is 0% across all models.
For an AI assistant to become part of city life, giving directions or guiding visually impaired people, it first has to be evaluable in real street scenes. But existing outdoor benchmarks each miss something. 3D simulators (CARLA, EmbodiedCity) are neither photorealistic nor structurally complex; Google-Street-View environments (StreetLearn) look real but have no dynamics and are discontinuous between panoramas, so the agent cannot walk continuously; video-to-sim work (Vid2Sim) uses short clips that cannot support district-scale exploration. What is missing is a district benchmark that is photorealistic, dynamic (pedestrians and vehicles), continuously walkable, and ground-level first-person.
The environment is the authors' earlier Realistic Virtual World: 602 360-degree video segments cover 85 streets of Akihabara in Tokyo, projected onto a sphere and stitched in Unity into a walkable pose graph of 193 nodes, 305 edges, mean branching degree 3.16, roughly 750 m north-south and 650 m east-west. The agent moves along prerecorded trajectories; it cannot teleport freely or physically interact (boundary discontinuities can occur, but the authors report such actions are only 11.3% and not overrepresented in failures).
The 175 tasks were hand-crafted in Unity by eight annotators who had personally visited Akihabara, taking about 60 hours, across three categories and seven subtypes (25 each):
Each task is labeled Easy/Medium/Hard by path length, instruction ambiguity, and landmark visibility, not by fixed step counts. Four evaluation protocols: exact match (localization), fuzzy match (a GPT-5 judge, agreement with human majority kappa = 0.937), coordinate match (final position within 10 m, or 20 m for map navigation), and mean relative accuracy (object count).
Agents use a ReACT loop with reflection memory and seven discrete actions, across six large multimodal models (GPT-5, Claude Sonnet 4.5, Gemini 2.5 Flash, Qwen2.5-VL-32B, InternVL3.5-8B/38B).
The headline number: the strongest model, Gemini 2.5 Flash, scores 17.1% overall against 77.3% for humans.
Several specifics. Map navigation is 0% for every model (humans 92%), the hardest task, unreachable even with a top-down map. Image-goal landmark search beats language-goal: GPT-5 scores 48 on the image variant versus 16 on language, because an image carries appearance, texture, and surrounding context that text cannot (InternVL is the exception and does not benefit). Performance drops steadily with difficulty. Providing a map of the agent's current location does not consistently help and sometimes hurts (relational reasoning falls 48% to 32%), which means aligning the map position with the first-person view is itself hard for these agents.
Failure modes differ by model. GPT-5 rarely makes low-level action errors but stalls on poor exploration strategy (55% of failures); Gemini and InternVL mostly fail on low-level control (around 40%); in spatial reasoning, perception failures surge across all models (reaching 38% for Gemini), and missing small visual details is fatal.
It turns the vague question of whether embodied agents are any good in real cities into a reproducible, decomposable quantitative exam. Photorealism alone is not enough; you also need dynamics (people and vehicles), continuous walkability, and a first-person view, and this environment has all four. For people working on multimodal models and embodied intelligence, its diagnostic value is that it tells you whether a given model loses on perception, grounding, exploration, or planning, not just that it is bad. The underappreciated weakness of map-to-visual alignment deserves to be studied on its own.
The authors' own caveats: only one district (Akihabara) so far, so cross-regional generalization cannot be claimed; the agent walks only along prerecorded trajectories, cannot roam freely, and has no physical interaction (though the data shows boundary actions are not overrepresented in failures).
One concern. The human baseline is local experts who have been to Akihabara, not average people, which effectively gives humans a domain-knowledge boost, so 77.3% is a high upper bound and the gap to an average person may be smaller on paper. That does not affect the main conclusion that the models are still bad. Also, 175 tasks with 25 per subtype is a small sample, so per-task scores (especially the single-digit ones) have wide confidence intervals; the ranking is more trustworthy than the absolute numbers.