Luna-TTS Family Technical Report
Feng Yin, Shuai Shi, Junjie Zheng, Kechenying Zhou, Yiqiu Wang, Chenyang He, Qiuhua Jiang, Mengxiao Bi, Yanmin Qian, Mingxin Chen, Xun Gong, Tianteng Gu, Bing Han, Peng Jiang, Chenda Li, Haiyang Sun, Han Wang, Wei Wang, Yi Wang, Leying Zhang, Wangyou Zhang, Chushu Zhou
cs.SD, eess.AS
2026-08-12
VUI Labs pretrained a diffusion-language-model TTS family on 1M hours of speech, topping all four Seed-TTS-Eval metrics while its streaming variant delivers first audio in 41.6ms.
Most production TTS systems today are autoregressive codec language models: they generate speech tokens strictly left to right, the same way a text LLM generates words. That recipe inherits LLM-scale training infrastructure and works well, but it carries real structural baggage. Latency grows linearly with utterance length, a single wrong token early on propagates through the rest of the generation (exposure bias, showing up as skipped words and repetitions), and RVQ (Residual Vector Quantization) tokenizers stack multiple codebook layers per audio frame that have no inherent left-to-right order, forcing AR models to impose an artificial one. The field has patched around this with interleaved codebooks and hierarchical schemes. VUI Labs' report tests a different path: building TTS on diffusion language models instead, to sidestep these constraints directly.
Luna-TTS Family consists of two variants sharing one tokenizer, one data pipeline, and a 0.6B-parameter backbone, both obtained by progressively adapting a pretrained autoregressive text LLM (Qwen3-0.6B): first from causal to bidirectional attention, then to block-causal attention.
Luna-TTS is fully non-autoregressive: it's trained with unrestricted random masking over the entire RVQ token grid and decoded through confidence-based parallel iterative sampling, generating the whole grid for an utterance in a fixed number of refinement steps, with a separate Qwen3-0.6B-based duration predictor supplying the target length. Because generation is framed as any-order infilling, zero-shot voice cloning and speech editing come for free within the same framework, with no task-specific machinery needed.
Luna-TTS Realtime is derived from Luna-TTS by continual training on a block-diffusion objective: causal across blocks of 32 codec frames (1.28s), but denoising all frames and codebooks within a block in parallel. It supports KV-cached blockwise generation, streaming audio out as soon as the first block commits.
After pretraining, an annealed fine-tuning stage adds explicit control over emotion and non-verbal vocalizations such as laughs and sighs, and a reinforcement-learning stage applies GRPO, with policy ratios computed over the denoising trajectory actually realized, rather than a left-to-right factorization as in prior AR TTS RL work.
On Seed-TTS-Eval, Luna-TTS ranks first on all four metrics: 0.73 CER on Mandarin (versus 0.83 for MiniMax-Speech), 1.49 WER on English (versus 1.54 for Qwen-Audio-3.0-TTS), and the highest speaker similarity on both languages (79.7 zh, 76.8 en). On the harder in-the-wild CV3-Eval benchmark, it posts the lowest Mandarin and English error rates (3.17 CER, 3.18 WER), with Korean as its weakest language (5.93 CER). The streaming variant, Luna-TTS Realtime, preserves most of that quality while running truly streaming: under a warmed local serving protocol it reaches an end-to-end RTF of 0.0240 and commits its first 1.28s audio block in 41.6ms, over 40x real time. It only trails the offline model by about 0.3 CER/WER and 3 similarity points on regular content, but the gap widens sharply on hard subsets with irregular text and noisy prompts (Mandarin hard-subset CER jumps from 6.90 to 12.56), because block-causal decoding commits each block irrevocably and can't revise an early mistake the way the offline model's global refinement can. On expressive-control evaluation, benchmarked against three commercial systems (MiniMax Speech 2.8 HD, ElevenLabs Eleven v3, Fish Audio S2.1 Pro), Luna-TTS wins on most objective, model-based, and human-rated metrics for both emotion and non-verbal vocalization control.
This is the largest diffusion-language-model TTS pretraining effort to date (1M hours of speech), and the first to push this architecture to four languages (Chinese, English, Japanese, Korean) while matching or beating leading commercial AR systems on public benchmarks. What's practically useful is that Luna-TTS and Luna-TTS Realtime share one tokenizer, data pipeline, and backbone lineage, so choosing between them is purely a deployment decision, not a system migration: pick the fully parallel model for offline throughput, or the block-causal model for low-latency streaming. That's a signal that diffusion-based TTS is no longer just an experimental alternative to AR decoding; at production scale it's now competitive on the metrics that matter.
The authors flag that Korean, at 6.9% of the training mixture, is the weakest of the four supported languages on CV3-Eval. Language coverage tops out at four, narrower than massively multilingual TTS systems. Luna-TTS relies on an external duration predictor to set the target frame count before synthesis, while Luna-TTS Realtime uses a fixed 1.28s block size chosen for streaming efficiency rather than adapted to content, which is part of why it trails most visibly on the hardest CV3-Eval subsets. The quality gap for the streaming variant is concentrated in the combination of irregular text and degraded reference audio; on regular content the cost is small, so the trade-off depends heavily on how much tail-case robustness a given application needs.