From-scratch TTS pipeline: FastSpeech2 + HiFi-GAN hits 8.8% WER on LJSpeech

Ashkingz · reddit · 2026-09-07

A data scientist built a TTS pipeline from scratch to truly understand the internals: CTC forced alignment → FastSpeech2 acoustic model → standalone PostNet → fine-tuned HiFi-GAN vocoder, trained on LJSpeech-1.1. Code and weights are open-sourced.

Results (100 val utterances, Whisper base.en scoring):

WER: 8.8% with PostNet vs 14.5% without; CER 5.1% vs 8.2%; MCD 7.02 vs 7.54 dB; RTF 0.080
Ground-truth floor through the same Whisper scorer is 4.9% WER; manual review attributes the remaining gap mostly to digit transcription and proper-noun errors, not synthesis failures

Highlight: natural pauses

Instead of a separate pause model, <sil> tokens are inserted at CTC alignment gaps, gated on duration (≥6 frames, 120ms) AND energy (mean RMS < 5% of utterance max) — the energy gate matters because stop-consonant closures are genuinely silent for 100+ ms

Debug war story

Fine-tuning HiFi-GAN generator from the official LJV1 checkpoint with randomly initialized discriminators produced shrill, metallic output; pulling discriminator weights from the separately-released universal checkpoint fixed it, with dloss stabilizing at 1.6–2.2

Known limitation: phoneme-based operation handles unseen words but chokes on unfamiliar phoneme sequences — e.g. "Higgledy-piggledy" comes out audibly stretched because the duration predictor lacks a prior for that pattern.

Original post →

From-scratch TTS pipeline: FastSpeech2 + HiFi-GAN hits 8.8% WER on LJSpeech

More from Multimodal