Confucius4-TTS: Transcript-Free Cross-Lingual Zero-Shot TTS with a Learnable Speaker Encoder
Huaxuan Wang, Huimin Wang, Ruiyu Zhang, Yingjie Li, Yitao Duan
cs.SD, cs.CL
2026-08-12
NetEase Youdao open-sources a 14-language zero-shot TTS that clones voices without reference transcripts via a jointly trained speaker encoder, hitting 3.73% average WER on CV3-Eval cross-lingual cloning and the best overall human-evaluation rank on an internal set.
The standard recipe for zero-shot TTS conditions on a reference audio plus its transcript, treating the reference's speech tokens as a prefix the model continues from. That design stalls in the wild: reference audio from podcasts, interviews, dialects, and low-resource languages mostly has no transcript, and a bad transcript drags cloning quality down with it. Video dubbing, audiobooks, and accessibility tools, the settings that need cross-lingual cloning most, are exactly where transcripts are scarcest.
Several transcript-free routes exist: Cross-lingual F5-TTS leans on MMS forced alignment, X-Voice on fine-tuning with synthetic audio prompts. Confucius4-TTS (NetEase Youdao) strips the auxiliary machinery away and uses only a speaker encoder trained jointly with the generator.
The system is two-stage. T2S (text-to-semantic) is a 24-layer decoder-only Transformer that autoregressively predicts semantic tokens, extracted by a frozen MaskGCT codec, from the target text. S2A (semantic-to-acoustic) renders those tokens into 80-dimensional mel-spectrograms with conditional flow matching on a DiT backbone, and a BigVGAN vocoder produces the waveform.
The transcript-free trick is how the speaker condition is supplied:
One model serves two inference modes. Reference cloning (the default) takes just the speaker embedding and target text, unconstrained by reference prosody. Continuation cloning, when a transcript exists, prepends it plus the reference semantic tokens for higher speaker similarity. S2A also consumes T2S hidden states rather than discrete tokens alone, softening the quantization bottleneck, and joint condition dropout during training enables classifier-free guidance at inference.
Training used roughly 500k hours of real plus synthetic speech across 14 languages (Chinese, English, Japanese, Korean, German, French, Spanish, Indonesian, Italian, Thai, Portuguese, Russian, Malay, Vietnamese), cleaned by source separation, multi-system ASR cross-filtering (error rate under 2.5%), and speaker clustering, on 32 A40 GPUs.
| Benchmark | Metric | Confucius4-TTS | Comparison |
| CV3-Eval cross-lingual (6 dirs) | avg WER/CER | 3.73% | CosyVoice 2 hits 48.10 on ja→zh |
| CV3-Eval, ja→zh | CER | 4.87% | CosyVoice 2 48.10, OmniVoice 52.64 |
| X-Voice cross-lingual (7 dirs) | lowest-CER directions | 4 of 7 | remaining three within 0.3 absolute points |
| Seed-TTS-eval English | WER / SIM | 1.49 / 0.700 | Qwen3-TTS 1.24 / 0.714 (with transcript) |
| Seed-TTS-eval Chinese | CER / SIM | 0.94 / 0.765 | Seed-TTS 1.12 / 0.796 (with transcript) |
| Internal human eval | overall rank | first on 3 of 4 directions | VoxCPM2 takes zh→en |
Most baselines in those tables carry a dagger, meaning they used a reference transcript at inference; Confucius4-TTS's numbers are transcript-free. In blind human evaluation it placed top-two on timbre similarity and naturalness across all four directions, while ElevenLabs (Eleven v3) kept first place on pronunciation for zh→ko and zh→ja.
The practical bottleneck in cross-lingual dubbing and global content production is data, not audio quality: usable reference audio often has no transcript. This route removes the mandatory ASR pass from the inference chain while keeping continuation mode in the same model for an extra notch of similarity when a transcript does exist. Code, checkpoints, and demos are on GitHub (netease-youdao/Confucius4-TTS), making it one of the few open systems with full objective-plus-subjective evaluation across a 14-language span.
For app developers the draw is multilingual coverage with a zero-dependency inference chain. For researchers, the dual-mode ablation is a clean reference point: continuation buys similarity at the cost of error rate (English 0.700→0.715 SIM, 1.49→1.68 WER).
The authors' own list: emotional expressiveness and fine-grained style preservation remain weak (first place on emotion in only one of four human-eval directions); long-tail language data and Chinese dialect coverage need work; streaming, low-latency inference and serving cost are future work.
Points to watch when reading. The share of synthetic data inside the 500k hours is never quantified beyond "a small portion." ElevenLabs numbers on MiniMax-MLS-Test are copied from the MiniMax paper rather than re-run, so evaluation conditions may not be perfectly aligned. The internal human-eval set is not public and cannot be reproduced. Russian WER of 4.64 sits mid-to-low among all comparisons with no explanation. SIM trails Seed-TTS and VoxCPM2 (both in transcript-using configurations) across the board, which says the transcript-free mode still pays a structural cost in timbre fidelity, exactly the hole continuation mode exists to patch. And with 25 Euler steps plus CFG at inference, latency and cost comparisons have no numbers attached.