CTC-TTS: LLM-based dual-streaming text-to-speech with CTC alignment
Hanwen Liu, Saierdaer Yusuyin, Hao Huang, Zhijian Ou
eess.AS, cs.AI, cs.SD
2026-02-23
CTC-TTS replaces MFA with CTC alignment and bi-word interleaving, cutting single-speaker streaming WER from LLMVox's 2.40% to 1.50%; zero-shot continuation MOS reaches 4.33.
LLM-based TTS can already turn text into discrete speech tokens and decode a waveform. Most of those systems still wait for the full sentence before they speak, so first-packet latency is too high for dual-stream use, where text arrives while audio is already going out.
Two bottlenecks sit in the way. Alignment usually comes from Montreal Forced Aligner (MFA), a GMM-HMM forced-alignment toolkit: heavy pipeline, retrained whenever the data changes. Sequence layout is the other one. Fixed-ratio interleaving of text and speech tokens ignores how speech actually lines up with words, so the model struggles to learn local phoneme-to-token maps. Alignment-aware interleaving helps, but it typically still depends on MFA, and papers disagree on the unit (word, BPE, or phoneme).
CTC-TTS replaces MFA with Connectionist Temporal Classification (CTC), which aligns sequences without frame-level boundary labels. A 115M Whistle ASR trained on LibriSpeech emits phoneme posteriors at 25 frames per second. Viterbi finds the most likely path; blank frames are assigned to the next phoneme. WavTokenizer, a single-codebook neural audio codec, emits tokens at 75 frames per second, so each phoneme maps to three speech tokens. The alignment is structural, not frame-accurate. That is enough to build word-level phoneme–speech blocks for an autoregressive model.
The interleaving unit is a bi-word block: phonemes of the current word, the separator, phonemes of the next word, then the speech tokens of the current word, closed by an end-of-block token. The extra word is there because pronunciation depends on neighbors.
Two variants split the quality–latency trade-off:
The loss is cross-entropy on speech tokens and the end-of-block symbol only. Single-speaker models use 4 Transformer layers; the multi-speaker zero-shot models use 12 layers and about 160M parameters.
On single-speaker streaming with VoiceAssistant400K (about 1,750 hours of training), both variants beat LLMVox, which uses fixed-ratio feature stacking. Naturalness is tied; intelligibility is not:
| Method | WER↓ | CER↓ | First-packet latency↓ | UTMOS↑ |
| LLMVox | 2.40 | 1.36 | 167 ms | 4.15 |
| CTC-TTS-F | 1.80 | 1.04 | 159 ms | 4.15 |
| CTC-TTS-L | 1.50 | 0.79 | 210 ms | 4.15 |
F is slightly faster than LLMVox. L waits about 50 ms more and cuts another 0.30 WER points.
On zero-shot continuation after 960 hours of LibriSpeech, CTC-TTS-L reaches WER 4.82 and MOS 4.33, against a reproduced MFA+ELLA-V baseline at WER 10.98 and MOS 3.94. Ablations split the credit: with the same bi-word layout, CTC slightly beats MFA (MFA+bi-word WER 5.14); with the same CTC aligner, bi-word far beats ELLA-V local-advance (CTC+ELLA-V WER 12.01). The gap widens on cross-speaker Seed-TTS test-en: MFA+ELLA-V hits WER 34.89, CTC-TTS-L 6.33, with MOS 4.23 vs 3.75.
For anyone shipping streaming TTS, the paper separates two engineering problems: you can drop the MFA pipeline, and you do not have to guess a fixed interleave ratio. Pick L or F from the latency budget. The ablations say most of the gain is in the bi-word layout; CTC is the cheaper aligner, not the whole story.
This is incremental. The backbone is still a single-codebook autoregressive Transformer, English-only. Teams already on CosyVoice2 or SyncSpeech should look at the interleaving unit. Teams without an MFA stack get a cheaper on-ramp.
The aligner is an English Whistle model trained on LibriSpeech, and phonemes come from Phonetisaurus WFST G2P. Other languages and messy spellings will break at G2P first. There is no head-to-head with multi-codebook codecs. Zero-shot tests cover LibriSpeech continuation and Seed-TTS test-en only: no long-form, no emotion, no noise. The authors flag neural G2P and tighter neural forced alignment as future work.
First-packet latency assumes the full text is already available. In a real dual stream, text arrives word by word, and L still waits for the next word, so end-to-end delay will exceed the reported 210 ms. The paper does not report RTF or GPU throughput, and it does not compare against SpeakStream or SyncSpeech on the same split.