Omni Interaction Agent Technical Report
Orantqing, Shengpeng Ji, Junlong Tong, Jialong Zuo, Dongjie Fu, Di Cao, Yangzhuo Li, Shangda Wu, Franz, Evan, Theron Veyra, Changhao Pan, Jingyu Lu, Dongchao Yang, Zhifei Xie, Yang Tan, Xiaoyu Shen, Xiaoda Yang, Wenfu Wang, Teddysun, Steveyves, Zhou Zhao, Bryanytian
eess.AS, cs.AI, cs.LG, cs.MM, cs.SD
2026-09-09
A 9B cerebellum runs full-duplex chat; a Codex brain takes long jobs. It takes every turn and interrupts 8%, but Pass@1 is 0.400 vs GPT-Realtime 0.600.
Most speech and omni models still run turn by turn: an external VAD cuts the user's utterance, then the model speaks. People do not talk that way. They barge in, drop a short "yeah," decide whether to answer in noise or in a group, and they revise a running task while chatting. If timing lives in an add-on module, the model never learns those calls. If casual chat and long coding jobs share one large net, latency and depth pull against each other.
Gander, from Tencent's Hunyuan speech team, puts full-duplex audio-visual conversation and agent workflows in one system. The front stays fast. The back can be slow.
The front "cerebellum" is 9B, built on MiniCPM-o 4.5's Thinker-Talker. Vision uses SigLIP, frames capped at 448×448, about 16× compression versus the raw patch grid. Audio starts near 50 frames per second, then an MLP downsamples 5× to about 10 tokens per second. Time is cut into one-second chunks. Each chunk concatenates the current audio-visual tokens, one control token, and whatever text the model emits. The control token is listen, speak, or interrupt. Context is a 128-chunk sliding window, roughly two minutes. The LLM does not emit acoustic units. Hidden states feed a small speech-token decoder, then a streaming flow-matching vocoder; timbre follows a reference clip in the system prompt.
The back "brain" is training-free, default Codex, GPT-5.6 in the reported evals. The cerebellum hands off with taskstart, tasksend, and taskresolve. A runtime owns streams, permissions, and delivery. The brain sees ASR text plus last relevant video frames, not the raw multimodal stream. Training data is about 2.7M examples: 37% speech interaction, 41% audio-visual, 13% agentic, and the rest noise, multi-party, and stay-silent negatives.
On Full-Duplex-Bench v3 (100 scenarios), Gander takes the floor in 100% of cases and starts too early in 8.0%, versus 13.5% for GPT-Realtime and 33.0% for a Whisper-GPT-4o-TTS cascade. Task scores lag: Pass@1 0.400 against GPT-Realtime's 0.600; ToolSel 0.759 and ArgAcc 0.503, a shade under Ultravox's 0.794 / 0.513. Driving the brain from text only lifts Pass@1 to 0.520 and ToolSel to 0.934, so the execution tier is not the bottleneck. The 51.6% filler rate is explained as holding the floor while work sits in the brain.
On SpokenQA, Llama Questions is 75.60 and Web Questions 59.30, ahead of Audio-Interaction's 67.31 / 54.34 inside the full-duplex group. WorldSense 49.62 and Daily-Omni 78.53 sit below the MiniCPM-o 4.5 init at 55.70 / 80.20. Audio plus video beats the better single modality by 8.88 points overall.
The product shape is "voice GPT plus a background Codex": small talk stays local, long jobs escalate. A 9B front already matches commercial duplex systems on turn-taking, which is evidence that interaction state can be an autoregressive target rather than a VAD rule. Weights and code are released; the industrial line points at a larger Hy-Realtime model.
Task accuracy is clearly behind, with Pass@1 near the weakest baseline. WorldSense drops 6 points from the init, so interaction training taxes static omni understanding. The brain only sees ASR text and last frames, so a mid-task correction or a pointed "this one" can arrive already wrong. The authors say on-policy distillation and RL are still missing, and the two tiers still talk through transcripts rather than a structured channel. Whether the filler rate is a holding strategy or an inability to go quiet is not ablated.