OmniEcho: Audio-Visual Spatial Understanding for Omni-Modal Embodied Agents
Ruixun Liu, Yuxuan Wang, Jiacheng Xie, Yuhuan You, Donghua Cai, Junming Lin, Xiong-Hui Chen, Zhifang Guo, Yunfei Chu, Qize Yang, Xize Cheng, Jin Xu, Yiwu Zhong
cs.SD, cs.AI
2026-09-20
OmniEcho grafts an FOA spatial encoder onto Qwen3-Omni, reaching 28.5 overall on real spatial QA and 16.2% SR on sound-guided nav, near some text VLN baselines.
Embodied agents still hear like a cheap microphone. Omni-modal models such as Qwen3-Omni take mono audio well enough to name a sound, but they do not know where it came from. Spatial hearing is how people find a speaker behind a wall or a crash off camera. Collecting real first-order ambisonics (FOA) with matched video and poses is expensive, and naive simulation often breaks geometric consistency among source, camera, and trajectory.
OmniEchoBench records 197 real indoor scenes across 30 environments, with 2,972 QA pairs and 900 navigation samples. QA covers source direction (1,069), 3D localization (521), source motion (473), camera rotation (309), and a four-choice cognitive-map task (600). Navigation supplies FOA plus a text instruction so text-guided VLN methods can sit on the same scenes.
Training data is rendered with a pipeline that keeps sources, views, and agent paths on one geometry. Existing VLN trajectories (ScaleVLN, RxR, R2R, VLN-PE) get a source at the goal, plus distance falloff and reverberation. Stage-3 instruction tuning uses 363,193 examples on 32 A100s for about four days.
The model starts from Qwen3-Omni-30B-A3B. A light FOA encoder takes a 5-channel map (W-channel log-mel plus three active-intensity axes and diffuseness), aligns to a frozen CLIP text encoder with SigLIP, then learns query-conditioned localization. At graft time the FOA encoder, native audio tower, and vision tower stay frozen. Only the LLM and a projector train. The W channel still feeds the semantic audio path; spatial tokens are inserted right after the matching audio tokens.
On audio-visual QA, OmniEcho scores 28.5 overall against 18.5 for the same backbone with mono audio plus vision. Audio-only is 21.8 versus 17.5. Dropping vision barely moves direction, localization, and motion, while camera rotation falls from 41.7 to 29.1. FOA carries source geometry; camera-centric questions still need pixels.
| Method | Setting | Overall |
| Qwen3-Omni | mono + vision | 18.5 |
| SO-7B | FOA + vision | 11.4 |
| OmniEcho | FOA + vision | 28.5 |
On real sound-guided navigation, success is 16.2% and SPL 11.5%, close to InternVLA-N1's 17.8 / 16.7 under text instructions. On VLN-CE R2R with simulated audio, success is 22.2%, far behind InternVLA-N1 at 50.6%. Freezing the FOA encoder in stage 3 scores 28.5; unfreezing it drops to 26.2. Localization error is 17.9° / 7.70° / 1.13 with 5-channel features, better than raw four-channel waveforms.
Grafting a spatial pathway onto a pretrained omni model is cheaper than training a navigation stack from scratch. Indoor search-by-sound is the realistic use. Sound-guided navigation is not yet a replacement for strong text VLN.
Almost all training audio is simulated; the paper flags distribution bias and only hints that a little real FOA might correct it. 3D localization remains 14.2. Navigation comparisons are imperfect: SoundSpaces consumes binaural audio, and text VLN gets route-level language. The SOTA claim holds on a self-built benchmark. Code and data are still marked as forthcoming.