0.6B WaveGen hits Seed-en WER 1.47 and SIM 0.75 with no vocoder

WaveGen: Truly End-to-End Waveform Generation via Internal Spectral Trajectory Learning

Sang-Hoon Lee, Ha-Yeong Choi

NeurIPS 2026

cs.SD, cs.AI, eess.AS, eess.SP

2026-11-09

WaveGen, a 0.6B diffusion model, maps text to 24 kHz audio by learning an internal mel path plus the waveform. Seed-en WER 1.47, SIM 0.75; no codec, vocoder, or duration model.

What problem this solves

Zero-shot TTS usually stops at an intermediate representation. Codec models predict discrete speech tokens, mel pipelines predict a spectrogram and pass it to a separate vocoder, and latent diffusion inserts a pretrained compressor. Extra modules split training into stages and cap the waveform at whatever the previous stage can express. VITS can emit a waveform directly, but it depends on a GAN and variational inference, and it was not built for zero-shot prompting. E3-TTS does run diffusion on the waveform. The text side still uses BERT, and the sequence is expensive.

WaveGen tightens the recipe to one model and one training run: text in, a 24 kHz waveform out. There is no codec, mel vocoder, or VAE, no standalone duration predictor, and no BERT, Wav2Vec 2.0, or semantic distillation. Alignment is the hard part. At full waveform resolution, plain conditional flow matching (CFM, a velocity field that carries noise onto the data) does not lock text to speech.

Method

The 24 kHz waveform is reshaped into 100 Hz frames, and a Diffusion Transformer (DiT) matches the flow on those frames. Text embeddings are average-pooled and stretched to the speech length. The smaller models do not attach a pretrained text encoder. Training is masked infilling: the prompt stays visible, and the loss covers only the masked span.

A waveform target alone does not align. Spectral Forcing adds a second target in the same network, a 64-bin mel used only as internal frequency guidance. Wider bins pull capacity toward reconstruction. On a 113M LibriTTS model, dropping the guide gives WER 123.67 and SIM 0.19. At 256 bins, WER is still 126.50. At 64 bins, WER is 4.96, SIM is 0.47, and DNSMOS, a neural quality score, is 3.72. Linear spectrograms diverge to NaN. Once the data gets large, the model leans on the spectral path and alignment arrives late, so a two-layer MLP of width 32 and dropout 0.2 stays on at inference too.

Shared DiT blocks align the sequences. Shallow separate heads refine waveform and spectrum. Eighteen shared layers and no separate head score WER 7.83. Fourteen shared layers plus two heads reach 4.96. Four heads fall back to 7.64.

The network predicts the clean waveform and the clean spectrum. That is x-prediction. The loss compares velocities, v-loss: the clean prediction is converted to a velocity and matched against the velocity of the interpolation. Optimizing the clean sample directly produces metallic noise, reported only qualitatively. Predicting velocity instead reaches WER 3.93 and SIM 0.48, but DNSMOS falls to 2.94, against WER 5.57 and DNSMOS 3.45 for x-prediction.

A multi-scale spectral v-loss sends the waveform velocity error through six STFTs, bins from 512 down to 16, normalized by ground-truth energy, with weight 0.1. A direct Flow2GAN-style spectral loss drives WER to 68.82. The multi-scale term alone raises WER to 16.98. Added to the waveform and spectral velocity losses, WER is 4.96 and DNSMOS is 3.72.

Sampling takes 32 Euler steps. Classifier-free guidance is 2, and the text condition is dropped only for t below 0.02. Extending that dropout to 0.5 hurts similarity. The paper does not say by how much. Each step also rewrites the prompt region by interpolating the real prompt with noise, so the sampler cannot drift off the prompt.

The 0.6B numbers that get quoted use a different front end: a ConvNeXt text encoder and filler tokens that pad text out to the speech length, with hop 480 at 24 kHz, or 50 Hz.

Results

On Seed-en, WER is Whisper's and SIM is WavLM-Large's. Baseline parameter counts omit the external decoder.

MethodSizeWER↓SIM↑
Ground truth-2.140.73
F5-TTS0.3B2.000.67
ZipVoice0.1B1.700.69
CosyVoice3-RL0.5B1.680.69
Qwen3-TTS0.6B1.680.70
Qwen3-TTS1.7B1.500.71
VoxCPM0.5B1.850.72
WaveGen-S0.1B4.420.53
WaveGen-B0.4B1.770.63
WaveGen0.6B1.470.75

The 0.6B model, one stage on about 100k hours of public English and Chinese audio, reaches WER 1.47 and SIM 0.75, a notch above ground-truth SIM 0.73. S and B use about 120k hours of English and land at WER 4.42 and 1.77. Qwen3-TTS and VoxCPM do not report data size in the table, so those rows are not a locked data match. WER below the recorded 2.14 mostly means the transcriber is lenient on synthetic speech.

The listening test includes only S and B. WaveGen-B naturalness is 4.000±0.126 against 4.117±0.112 for ground truth. Similarity is 3.988±0.037, overlapping F5-TTS at 3.989±0.037 and Qwen3-TTS-1.7B at 4.008±0.037. The listening test skips the 0.6B model.

On the smaller matched-architecture setup, a 113M vocoder given ground-truth mel scores WER 1.47, SIM 0.70, DNSMOS 3.32. A 226M two-stage system at 32 steps total scores WER 5.17, SIM 0.47, DNSMOS 3.59. End-to-end at 210M and 32 steps scores WER 4.68, SIM 0.50, DNSMOS 3.71, RTF 0.12. Give the cascade 64 steps and WER falls to 4.39, with SIM 0.46, DNSMOS 3.53, and RTF 0.16. On that same small corpus, 407M end-to-end WER is 2.91. On an A100 with no inference kernel work, RTF is 0.08 at 113M and 0.24 at 407M. RTF is generation time divided by audio duration.

WaveGen-B at 4 steps has WER 52.0, at 8 steps 4.85, and at 32 steps 1.77 with SIM 0.63. At 100 steps, WER is 1.84 and SIM is 0.64.

On public Emilia, English plus Chinese, the 680M model with spectral dropout 0.2 and prompt anchoring scores Seed-en WER 1.47, SIM 0.75, UTMOS 3.68, and Seed-zh CER 1.75, SIM 0.80, UTMOS 2.87. UTMOS is a second neural quality score. In that same table, ZipVoice is at 1.70 / 0.69 in English, F5-TTS at 1.83 / 0.67, and WavTTS at 1.50 / 0.65. English error and similarity lead. Chinese CER is worse than ZipVoice at 1.40 and F5-TTS at 1.56, while SIM 0.80 is above ground truth at 0.76. Spectral dropout 0.5 without anchoring can reach English WER 1.39 with SIM only 0.70. The reported point is 1.47 and 0.75.

REPA, alignment to Whisper-small features, was tried only on the small LibriTTS model. WER moves from 4.96 to 3.54 and SIM from 0.47 to 0.54. The main models leave it out.

Why it matters

The saving is the cascade. One model aligns and synthesizes, so there is no codec, duration predictor, or GAN vocoder to keep alive. Training uses public Emilia, runs in one stage, and the code and checkpoints are released. Scale shows up: WER goes from 4.42 at 0.1B to 1.47 at 0.6B, after the prediction space, the spectral bottleneck, and the split heads are in place.

The English margin over Qwen3-TTS and VoxCPM is modest. Sampling is still 32 steps, and the 0.4B model has RTF 0.24 on an A100. Useful as a research baseline with fewer moving parts. Chinese intelligibility does not lead the same comparison.

Limitations

Runs other than 0.6B lasted about four epochs, 1.1M steps on four GPUs. The 0.6B model saw eight GPUs and 11 epochs. Longer training was not measured. REPA helps the small model and was turned off on the large one to avoid self-supervised features, so a known gain is unused. 48 kHz is not in the paper.

Emilia is MP3. Models trained on it have trouble above 8 kHz, while the same method on LibriTTS WAV keeps more high-frequency energy. An end-to-end model has to learn fidelity itself, and compression in the corpus becomes the ceiling.

The 1.47 / 0.75 result uses ConvNeXt, filler tokens, and a 50 Hz frame rate. Average pooling at 100 Hz is the small-model setup. Parameter counts in the main table omit decoders, and the listening test skips the 0.6B model. At a matched 32 steps, end-to-end WER is lower. At 64 steps the cascade reaches 4.39 and is lower. Prompt anchoring pins the prompt region to the real trajectory, and the paper never says whether SIM excludes that region. Chinese CER is not the best entry in the table. Metallic noise from the x-loss is not quantified.

Terms

Source

Related papers

All paper explainers