FireRedTTS3: Unified Speech Generation and Editing with Semantically Enriched Speech Representations
Feiyu Shen, Kun Xie, Yichen Wu, Ziqi Dai, Yichen Han, Junjie Li, Xuelong Geng, Fenglong Xie, Lei Xie, Xu Tang, Yao Hu
cs.SD
2026-08-18
FireRedTTS3 injects semantics into continuous latents via a frozen audio encoder. Base hits 3.04% avg error and 78.8% SIM on Seed-TTS-Eval; Instruct unifies design and editing.
Continuous autoregressive TTS models speech latents without quantization, so acoustic detail survives and the stack can still ride a text LLM's instruction following. The same unbounded latent space lets small one-step errors snowball into timbre drift and prosody collapse.
Prior fixes either bolt on extra semantic modules and multi-stage tokenizer recipes (Ming-UniAudio, VibeVoice, dots.tts) or squeeze the continuous features through an FSQ bottleneck (VoxCPM). Both work. Both make the system heavier. FireRedTTS3 puts the semantic constraint into the representation itself and keeps the generator as a plain LLM-DiT.
RedAE is a continuous tokenizer. A 24 kHz waveform is sliced into non-overlapping 480-sample frames (50 Hz), then two cascaded Qwen3-style Transformers pool that sequence to 25 Hz. Reconstruction is a GAN with X-Codec-style discriminators: adversarial, multi-scale Mel, feature matching, plus a semantic term.
The semantic teacher is the frozen Audio Encoder of FireRedAudio, itself trained on ASR and speaker verification. RedAE matches its 25 Hz latents to the teacher with MSE. After tokenizer training the teacher is thrown away. There is no KL term, on purpose: KL would over-compress acoustics. Training ran 550k steps on 32 H800 GPUs over 500k hours (50% clean speech, 25% noisy, 10% sound effects, 15% music).
The generator is LLM-DiT. An Aggregator pools 25 Hz latents into 6.25 Hz patches. A Qwen3-1.7B Backbone models text tokens and patches autoregressively. A DiT, conditioned on Backbone states via AdaLN, denoises each patch. Each step sees a noisy 4-frame current patch plus 12 frames of clean history. Classifier-free guidance drops the Backbone condition with probability 0.1 at train time.
FireRedTTS3-Base starts from Qwen3-1.7B-Base, prepends CAM++ speaker embeddings and a language tag, trains 170k steps on 2.6M hours of Chinese and English, then continues on 560k hours covering 24 languages and 21 Chinese dialects.
FireRedTTS3-Instruct starts from Qwen3-1.7B, uses ChatML, and keeps the text head. Instructions are first rewritten as a textual plan: 12 acoustic attributes for voice design, a target transcript plus an edit-region mask for editing. Stage two adds 330k hours of design and edit data for 40k steps. No speaker embedding, no language tag.
On Seed-TTS-Eval, Base posts the best average among reported systems: 3.04% WER/CER and 78.8% speaker similarity. Qwen3-TTS is 3.07% / 74.5%, CosyVoice3-1.5B is 3.06% / 75.3%, dots.tts (pretrain checkpoint) is 3.14% / 78.7%. SIM is 80.9% on Test-ZH and 77.2% on Test-EN.
On MiniMax-MLS-Test (24 languages) average error is 3.75% versus MiniMax-Speech at 3.77%; average SIM is 84.8% versus 83.5% for dots.tts. SIM is first or second in 22 of 24 languages. Portuguese and Ukrainian were not in the Audio Encoder or RedAE training mix; Ukrainian error still reaches 0.55%. Cantonese CER is high for every system; the paper blames Whisper-large-v3.
InstructTTSEval was scored with Gemini-2.5-pro after the official preview judge proved unavailable. Instruct wins all six APS/DSD/RP numbers: ZH 85.8 / 82.0 / 69.7, EN 80.7 / 82.3 / 72.0, against Qwen3-TTS-VD at 83.7 / 81.7 / 65.8 and 76.4 / 81.4 / 64.2.
Against Ming-UniAudio-Edit, semantic editing average WER is 6.97% vs 8.53% (ZH) and 10.22% vs 12.22% (EN); edit accuracy 87.27% vs 82.91% (ZH) and 78.91% vs 71.06% (EN). For acoustic edits, Chinese speed WER falls from 5.88% to 2.27%, and volume relative amplitude error from 14.9% to 3.58%.
| Setting | Metric | FireRedTTS3 | Strongest reported baseline |
| Seed-TTS-Eval avg | WER/CER ↓ | 3.04% | CosyVoice3 3.06% |
| Seed-TTS-Eval avg | SIM ↑ | 78.8% | dots.tts 78.7% |
| MiniMax-MLS avg | error ↓ | 3.75% | MiniMax 3.77% |
| InstructTTSEval-ZH RP | accuracy ↑ | 69.7% | Qwen3-TTS-VD 65.8% |
Semantic supervision can stop at the tokenizer. The generator does not need a second semantic branch or an FSQ bottleneck. A 1.7B backbone plus patch-level DiT is already enough to lead the public cloning, instruction-design, and editing tables in this paper.
The catch is data. RedAE alone took 32 H800s for 550k steps; Base stage one ate 2.6M hours. Simple architecture, not a small recipe. Code and weights are released.
There is no ablation that turns off the semantic loss. Error accumulation is the headline problem; the tables only show end-to-end scores.
Base and Instruct are two checkpoints, not one model doing three jobs. InstructTTSEval used a different Gemini than the official toolkit. Semantic editing has a single baseline. No MOS, no RTF. Cantonese numbers are contaminated by ASR, which the paper states.
Calling the architecture simple assumes FireRedAudio already exists as a teacher, plus million-hour speech. The bottleneck for a reimplementation is the corpus.