Motion-Omni: End-to-End Joint Speech and Full-Body Motion for Spoken Dialogue
Chengqian Ma, Wei Tao, Haoyu Zhang, Yiwen Guo
cs.SD, cs.CV, eess.AS
2026-08-29
Motion-Omni-Q7 emits full-body motion from speech hidden states, matching the same-audio LOM cascade within 2% at 5.4× lower latency (RTF=0.78) and 2.62% WER.
A talking avatar has to choose its words and its body motion together. Those skills currently live in different model families. Spoken dialogue models reply in audio with no body. Co-speech motion models move only after someone hands them a finished waveform. The usual product glue is a cascade: finish the dialogue model, then run a motion model on that audio. That costs a second full inference pass, and the motion loss never updates the speech or dialogue weights.
A few spoken-motion systems loosen this recipe. SOLAMI emits speech and body tokens, then paints the face afterwards. Ex-Omni jointly trains a 52-dimensional ARKit face decoder, face only. ViBES and MIBURI freeze the speech expert, so motion gradients never reach it. Motion-Omni asks whether an open-ended spoken dialogue model can emit face, hands, upper body and lower body as native outputs, with the motion objective allowed to update both the LLM and a distinct Speech Generator.
The reference instance, Motion-Omni-Q7, has four modules. A frozen Whisper-large-v3 encoder maps 16 kHz audio; a two-layer MLP projector packs every five frames into the LLM embedding space. The backbone is Qwen2.5-7B-Instruct. The Speech Generator, initialised from Qwen2.5-0.5B-Instruct, emits GLM-4-Voice units at 12.5 Hz over a 16,384-unit vocabulary. Four parallel part-aware decoders emit LOM VQ codes at 30 Hz for face, hands, upper body and lower body; a frozen LOM VQ-VAE decodes them into SMPL-X and FLAME.
The motion head never sees the waveform. It attends to the Speech Generator's last-layer hidden states as keys and values, and uses interpolated embeddings of the emitted speech tokens as queries, upsampled from 12.5 Hz to 30 Hz. Motion is grown from the same states that produce the speech. The four part decoders share that speech context and do not cross-condition on one another's sampled codes. The motion loss weights face:hand:upper:lower as 106:180:78:61, in proportion to the underlying feature dimensions.
Co-adapting speech and motion is required. A pilot that froze the speech pathway left the motion loss on a higher plateau, and the rendered motion was visibly off the audio. Training then proceeds in four stages: ASR on the speech projector only; TTS on the Speech Generator with the LLM frozen; TTSM with a 12.5/25/50/100% quality-quantile curriculum while the Speech Generator and Motion Generator train together, during which teacher-reference FGD falls from 0.3974 to 0.3040; then a Stage 4 mixture that unfreezes the LLM and mixes ASR, TTS, speech-to-speech-with-motion, and text-only dialogue.
Supervision is teacher pseudo-labels. LOM annotates consistent-voice replies from InstructS2S-200K. Samples are ranked by VQ reconstruction error and beat correlation, producing 422,856 pairs (1,402 hours). Training on the unranked full set diverged, which is why the quantile curriculum exists. Stage 4 also mixes about 213k text-only dialogues; without them the model more often echoed the user.
The evaluation set is the newly released SwDA-500: 500 open-ended prompts drawn from the Switchboard Dialog Act Corpus, covering all 66 topics. Motion comparisons hold audio fixed so that differences in wording and prosody do not confound the motion scores. BC and FGD are reported in ×10⁻¹ units.
| System | BC↑ | Diversity↑ | RTF↓ | Lip-sync LSE-C↑ |
| Motion-Omni-Q7 | 7.59 | 13.67 | 0.78 | 7.011 |
| Same-audio + LOM teacher cascade | 7.67 | 13.07 | 4.39 | 6.985 |
| Same-audio + EMAGE | 7.32 | 10.95 | 0.84 | 6.643 |
| Qwen2.5-Omni + LOM | 7.56 | 13.82 | 18.34 | 7.136 |
Among systems that do not run LOM at motion-inference time, Motion-Omni-Q7 has the highest beat correlation and diversity, and better rendered-video lip sync than the EMAGE and MambaTalk cascades. Against the same-audio teacher cascade, reference-free motion metrics land within 2%; a full speech-plus-motion response finishes in 4.32 s, 5.4× faster. Teacher-reference FGD is 3.03, the lowest among non-LOM-inference systems.
Spoken-dialogue ability survives the motion head. VoiceBench overall is 47.63, above LLaMA-Omni at 41.12 and Ex-Omni at 43.57. English WER on Seed-TTS-Eval is 2.62%, the lowest among the omni-modal LLMs compared, still above dedicated TTS (Qwen3-TTS-12Hz-1.7B at 1.24%).
A four-annotator preference study, 25 pairs per arm, three motion dimensions pooled: +25 vs same-audio EMAGE (45 wins / 10 ties / 20 losses); 25/27/23 vs the LOM teacher, with ties the most common verdict. The paper labels this exploratory and does not read the smaller margins as a stable preference.
If a talking avatar is still generated as "audio first, motion second", this paper offers a reproducible alternative: hang the motion head on the Speech Generator's hidden states, drop the second full inference pass, and turn alignment into a joint objective. The teacher and the modules are swappable; code and the pseudo-labeled set are public. For a team shipping a conversational avatar, this is closer to one model that talks and gestures than two pipelines that have to stay in sync.
The quality ceiling is still the LOM teacher, and the evaluation reference distribution is LOM's own. What the numbers show is narrower: near that teacher distribution, end-to-end can match the cascade and cut the latency.
Motion is bounded by LOM's VQ codebook and teacher pseudo-labels; motion outside that distribution cannot be expressed. Only the Q7 instance was trained. There is no speaker-identity or emotion conditioning, and the data is English only. The system consumes the full user utterance before emitting the first response token, so it is an offline response generator, not a streaming interactor like MIBURI.
FGD uses LOM's predictions on the same audio as the reference, which favours systems that look like the teacher. The human study has four annotators and 25 pairs per arm. A video LLM-as-judge correlated with people only on speech quality, so perceptual claims rest on the small human set. SwDA-500 is not a paired real-motion benchmark; whether an open-ended reply is accompanied by the "right" full-body motion still has no motion-capture ground truth.