Moshi: a speech-text foundation model for real-time dialogue
Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, Neil Zeghidour
eess.AS, cs.AI, cs.CL, cs.LG, cs.SD
2024-09-18
Kyutai's Moshi joins a 7B text LM, a 1.1 kbps codec, and dual audio streams. Theoretical latency 160 ms. Spoken Llama-Questions 62.3% vs SpeechGPT's 21.6%.
Phone-like voice assistants are still pipelines: VAD, ASR, a text LLM, TTS. Latency stacks into seconds. Emotion, accent, and room sound die at the text interface. Dialogue is carved into turns, so overlap, interruptions, and backchannels (10–20% of talking time) are out of scope. Human replies average about 230 ms across languages. Cascades miss that by an order of magnitude.
Moshi treats spoken dialogue as speech-to-speech generation. The model always listens and always emits audio (speech or silence). User and system occupy two token streams. There is no explicit "whose turn" switch.
Three parts. Helium is a 7B text Transformer trained from scratch: RMSNorm, RoPE, SwiGLU, 32k vocab, 2.1T English tokens. Mimi is a causal neural codec: 24 kHz in, 12.5 Hz out, 8 quantizers, 1.1 kbps. The first codebook distills WavLM semantics; seven acoustic quantizers run in parallel on a split RVQ so semantics and waveform do not fight over one residual. Adversarial-only training reaches MUSHRA 81.0 versus 58.8 with reconstruction losses, even though VisQOL goes the other way. The first 80 ms frame already encodes and decodes, so the stack can stream.
Generation is an RQ-Transformer: a large Temporal Transformer (initialized from Helium) steps at 12.5 Hz; a 6-layer Depth Transformer (width 1024) predicts the codebooks inside each frame. Acoustic codes lag the semantic code by 1–2 steps so the large model owns the semantic-acoustic dependence. Dual streams concatenate Moshi's and the user's codes at each step; at inference the user side is fed, the system side is sampled.
Inner Monologue inserts time-aligned text tokens (from Whisper) as a per-frame prefix to the semantic codebook. Text is denser; PAD is about 65% of those tokens. This is close to decisive for intelligibility and knowledge: without it, spoken QA drops to roughly a third. Delaying one modality relative to the other retargets the model. Audio delayed 2 s yields streaming TTS (4.7% WER on LibriSpeech test-clean vs Vall-E 5.9%). Text delayed 2 s yields streaming ASR (5.7% WER, behind a FastConformer at 3.6% with similar lookahead).
Training is staged: 1M steps of single-stream unsupervised audio, half the batches still text to limit forgetting; 100k steps of diarization-fake dual streams; 10k on Fisher phone calls; 30k of synthetic assistant scripts. Context in the experiments goes to 5 minutes.
Helium is competitive among 7B models with a similar compute budget: MMLU 54.3, above Llama 2 at 45.3, below Mistral 62.5 and Gemma 64.3, which used more compute. After audio training Moshi's MMLU is 49.7.
Spoken QA is where the cascade split shows. Inner Monologue lifts Moshi from audio-only WebQ 9.2 / LlamaQ 21.0 / TriviaQA 7.3 to 26.6 / 62.3 / 22.8, ahead of SpeechGPT 7B (6.5 / 21.6 / 14.8) and Spectron. Helium text topline is 32.3 / 75.0 / 56.4. TriviaQA is the painful gap; the authors blame oral-style instruction data that never sees multi-sentence syntax.
| Model | WebQ | LlamaQ | Audio TriviaQA |
| SpeechGPT 7B | 6.5 | 21.6 | 14.8 |
| Moshi, no Inner Monologue | 9.2 | 21.0 | 7.3 |
| Moshi | 26.6 | 62.3 | 22.8 |
| Helium text | 32.3 | 75.0 | 56.4 |
In self-play dialogues at temperature 1.0, DialoGPT perplexity is 79.3 versus 59.6 on real Fisher; overlap 4.1 s is close to the real 3.3 s. dGSLM's non-cascade baseline barely speaks (PPL 195.9). sWUGGY-style textless scores fall after instruction tuning; the authors distrust them relative to QA and turn-taking stats. 8-bit quantization costs about 2 MMLU points and halves size; 4-bit costs Moshi 5–10 points, worse than Helium.
This is the first full-duplex, streaming, knowledge-bearing spoken LLM that actually runs, with a public demo. Theoretical latency 160 ms, about 200 ms in practice, under the 230 ms cross-language human mean. Inner Monologue shows speech-to-speech can keep text as a per-frame semantic prefix, which stays streaming-friendly, unlike Chain-of-Modality that writes a full answer before speaking.
The license is CC BY-NC-SA 4.0, so not a commercial drop-in. The stack is heavy; reproducing it costs far more than a Whisper+LLM+TTS pipe.
The MMLU and TriviaQA gaps versus Helium show audio training crowding out factual capacity, and synthetic instructions not covering written syntax. Streaming ASR at 5.7% is explicitly not SOTA; they present it as a framework trick. Mimi's MUSHRA is high, generation quality under quantization is scored with MOSNet, and the two listening tests are never aligned on one table. Safety reports toxicity and voice consistency; the demo is still one voice. Acoustic degradation under quantization is harsher than for text LLMs, so endpoint recipes do not transfer. Dual-stream overlap is learned on Fisher, instruction tuning on scripts; there is no human-in-the-loop interruption study.