Ex-Omni-2D: Expressive Omni-Modal Dialogue Models with Native Visual Presence
Haoyu Zhang, Zhipeng Li, Xiaoying Tang, Tianshu Yu, Yiwen Guo
cs.AI, cs.CL, cs.CV
2026-08-11
Ex-Omni-2D makes an omni-modal dialogue model emit a reference-conditioned talking-avatar video alongside response text and personalized speech. A structured Visual Thought Plan turns dialogue context into scene, emotion, and motion intent, and a shared multi-codebook speech unit stream drives both speech synthesis and frame-aligned video; the 4-step four-GPU pipeline hits an end-to-end RTF of 1.293.
Omni-modal dialogue models understand speech, text, and images and reply with speech, but the reply is visually disembodied: a voice with no face. Existing speech-driven avatars typically need a hand-written auxiliary text prompt to control scene, emotion, and motion, which is inflexible in dialogue and tends to drift from the current conversational state. Mapping high-dimensional LLM hidden states straight into a video generator is expensive and model-specific, and large-scale data aligning query, text, speech, and video simply does not exist. The goal is to make one dialogue response produce text, personalized speech, and identity-consistent video without four-way paired data.
Ex-Omni-2D decouples the dialogue model from the video generator with two shared interfaces.
The first is the Visual Thought Plan (VTP). The dialogue backbone (Qwen3-8B) emits a structured five-field plan inside a constrained <thinking> block (first-frame scene, overall scene, emotion, movement style, motion detail) before generating the visible response. VTP is an inspectable textual intermediate, not an unconstrained chain of thought and not an opaque hidden-state mapping; it makes the high-level intent of how the response should look explicit.
The second is a native multi-codebook speech unit stream. The speech generator (built on Qwen3-TTS-0.6B) predicts 16 acoustic codebooks; the same units are decoded into a waveform and aligned to video frames (speech at 12.5 Hz, video at 25 fps, one unit block covering two frames) to drive avatar motion. This lets the speech and video pathways, which normally need different data, be supervised separately on speech, dialogue, and avatar-video data, then reconnected at inference by the VTP and generated units, sidestepping the need for four-way paired supervision.
The video generator has two forms. A full-sequence Teacher (Wan2.1-T2V-1.3B plus OmniAvatar weights) gives the best quality but is slow. For incremental deployment it is distilled into a block-causal Streaming Student, with Prefix Streaming: the previous chunk's clean latent is placed inside the current denoising window as an anchor, with stop-gradient and deduplicated caching so it is not regenerated. This targets the cumulative late-chunk degradation that plagues streaming generation.
On 200 VoiceBench/CommonEval speech queries, the Ex-Omni-2D Teacher reaches SIM 0.417 and Sync-C 4.95 while also producing the dialogue response, personalized speech, and visual plan. It does not top single rendering metrics (echomimic leads subject consistency at 97.87, OmniAvatar leads Sync-C at 5.64), but those baselines only render and do not converse.
The quality-efficiency trade-off is explicit. With the Teacher at 50 steps the E2E RTF is 26.917; the four-step Student reaches RTF 1.293 at 400x720 with SC 93.65, IQ 57.40, DD 32.00, Sync-C 3.90, and a first playable video chunk at 3.14 s. Eight steps push IQ, DD, and Sync-C higher at RTF 1.932. On speech QA it scores 4.28 AlpacaEval, 3.71 CommonEval, 58.70 BBH, second only to Qwen2.5-Omni among listed systems. On multi-turn OmniCharacter dialogue it averages 3.938 across fluency, coherency, and consistency, above its Qwen3-8B backbone's 3.537. Prefix Streaming cuts the last-minus-first chunk consistency error from 0.1005 to 0.0790, a 21.4 percent reduction, and is better on every chunk from 9 onward.
This adds the missing visual body to omni-modal dialogue along a deployable route: two lightweight, inspectable interfaces join speech and video pathways that had to be trained separately anyway, avoiding costly four-way paired data. Prefix Streaming suppresses cumulative streaming drift and gives a usable quality-efficiency point. Teams building avatars, virtual companions, or real-time voice assistants can borrow the decoupled architecture.
The authors concede several points. Speaker similarity to the reference still has room to improve. VTP is high-level semantic guidance, not an independently sufficient video-control signal; the final frame is jointly set by VTP and frame-aligned speech units, and a fixed text or audio CFG ratio does not transfer uniformly across responses. Generating VTP in the shared autoregressive channel introduces a measurable speech-QA and reasoning trade-off. Long-horizon analysis covers only temporal subject stability and facial-motion regularity. The Teacher remains expensive; the Student is faster but lower on IQ, DD, and Sync-C, with a 3.14 s first chunk and single-request E2E RTF above 1, so it is incremental output rather than end-to-end real-time interaction. One caveat for readers: the comparison reports each interface's best capability and is not a same-interface architectural ranking.