Tacit-TTS: From Autoregressive Decoding to Masked Prediction for Efficient Transcript-Free Voice Cloning
Jian Chen, You Zhang, Mark Vinton
eess.AS, cs.CL, cs.SD
2026-09-30
Tacit-TTS replaces IndexTTS2's autoregressive decoding with masked prediction and reflow distillation: near-teacher quality, 4.76x real time, no reference transcripts needed.
Zero-shot voice cloning has been a pick-two trade-off. Autoregressive systems like IndexTTS2 lead on quality and expressiveness, but they emit semantic tokens one at a time, run slower than real time, and are unusable for interactive use. Non-autoregressive systems such as F5-TTS and MaskGCT decode in parallel and are fast, yet most require a transcript of the reference clip at inference. When the reference is in a language the ASR front end does not support, or contains no words at all (infant babble, gibberish), the transcript comes back empty or hallucinated and the system fails at its input stage. NAR decoding adds a third constraint: the target duration must be fixed before generation starts.
Tacit-TTS, built at Dolby, is an 836M system distilled from IndexTTS2 that clears all three constraints at once: fast, transcript-free, and no upfront duration requirement.
The pipeline keeps the two-stage layout, text to semantic tokens (T2S) then semantics to audio (S2A), and trains only on teacher-synthesized data, never touching the teacher's original 55k-hour corpus.
Masked prediction for T2S. The teacher's 527M autoregressive Transformer is replaced by a 269M bidirectional DiT decoded MaskGIT-style: start from an all-[MASK] sequence, predict every masked position in parallel each iteration, commit only the most confident tokens, finish after a fixed 12 iterations. The forward-pass count is constant regardless of output length, which is what decouples latency from sentence length. Conditioning reuses the teacher's pretrained w2v-BERT + Perceiver encoders for speaker and emotion embeddings, and those encoders never needed a transcript in the first place. Text embeddings are upsampled to the target semantic length before fusion, keeping sequence length at T and acting as a builtup text-semantic alignment prior.
Training-free length estimation. Parallel decoding needs the target length up front. Instead of a learned duration predictor, the paper counts syllables with plain signal processing: RMS energy on the reference, voiced regions detected by YIN, energy peaks separated by at least 100 ms treated as acoustic syllables. Reference semantic frames divided by syllable count gives a pace factor, clamped to 7-12 codes per syllable, multiplied by the target text's syllable count (one per Chinese character; textstat for English). No parameters, nothing to retrain per language.
Two-head distillation. The training set is synthesized by the teacher: 874k utterances, about 2k hours, 2,739 reference speakers, each pairing a reference voice with content-independent target text. Only semantic codes and latent features are stored; the audio itself is not needed. One Transformer trunk carries two heads: a masked-prediction head for discrete semantic codes, and a residual-recovery head that reconstructs the continuous latent the renderer expects, so the downstream interface stays untouched. Mask ratios follow a cosine distribution, quantized into 16 buckets and injected via AdaLN.
Reflow for the renderer. The S2A stage keeps the teacher's flow-matching DiT and BigVGAN vocoder. It is first fine-tuned on the conditioning distribution produced by the new T2S model, then reflowed: run the 25-step solver, record (noise, mel) pairs, retrain on those couplings to straighten the trajectory. Euler steps drop from about 25 to 8 with little quality loss.
Four standard test sets, following the IndexTTS2 protocol:
| System | LibriSpeech SS/WER | SeedTTS-en SS/WER | SeedTTS-zh SS/CER | AISHELL SS/CER | Total speed (x real time) |
| CosyVoice2 | 0.843 / 6.00 | 0.794 / 3.28 | 0.846 / 1.45 | 0.834 / 1.97 | 1.03 |
| F5-TTS | 0.821 / 8.04 | 0.803 / 1.94 | 0.844 / 1.51 | 0.831 / 3.67 | 2.63 |
| Tacit-TTS | 0.875 / 5.54 | 0.849 / 2.34 | 0.829 / 1.71 | 0.804 / 2.21 | 4.76 |
| IndexTTS2 (teacher) | 0.870 / 3.12 | 0.860 / 1.52 | 0.865 / 1.01 | 0.843 / 1.52 | 0.81 |
Tacit-TTS posts the highest speaker similarity among deployable systems on both English sets, plus the best LibriSpeech WER. On both Mandarin sets it trails CosyVoice2, which the authors attribute to the 2,342-English versus 397-Chinese speaker imbalance in training.
Per-stage latency (roughly 5 s outputs, one A100): T2S drops from 5,274 ms to 199 ms, a 26.5x cut; S2A from 875 to 310 ms; generation overall from 6,149 to 509 ms, 12.1x. The speedup varies with length: 9.0x at 4 s, peaking at 14.9x near 30 s, easing back to 11.8x past two minutes as O(n²) self-attention in the NAR stages catches up. The "over 10x" headline holds only for outputs longer than 5 s.
The transcript-free advantage shows up under stress. With cross-lingual references in eight languages (including Yoruba and Hindi), transcript-dependent baselines blow out to 20-47% WER on English targets; CosyVoice2 fails on a third of references outright and MaskGCT fails on all Russian and Arabic ones. Tacit-TTS generates from every reference with 0.789 SS and 0.10% WER. On infant babble, transcript systems land between 31% and 99% WER while Tacit-TTS reaches 3.24% WER and 0.603 SS, beating the teacher's 0.484.
At 4.76x real time and 836M parameters, with top-tier English similarity, voice cloning now pairs above-real-time speed with competitive quality, which opens interactive agents and dubbing workloads. Transcript-free conditioning widens the usable input space: unlabeled archive audio, low-resource languages, and non-lexical sounds all become valid references, and transcript-dependent systems have no answer there.
The distillation recipe may matter more than the model. Two thousand hours of teacher-synthesized data, a two-head loss that preserves the continuous-latent interface, and reflow on the renderer moved a 1B-class teacher's capability into a roughly half-size NAR student without the teacher's corpus. Teams holding a strong AR TTS and needing to cut inference cost can copy this route directly.
Mandarin is the visible weak spot: both Mandarin sets sit below CosyVoice2 on SS and error rate, and the paper itself blames the speaker imbalance in training data. The gap to the teacher is not closed either; LibriSpeech WER is 5.54 against the teacher's 3.12, and perceptual quality (UTMOS/DNSMOS) is described as close, not equal.
The 10x headline is conditional: it holds above 5 s, is 9.0x at 4 s, and recedes for long outputs as quadratic attention cost accumulates.
Baseline quality numbers in the main table are quoted from the IndexTTS2 paper rather than re-run in a common environment (the metrics match, the runs do not); the cross-lingual and non-lexical baselines were run by the authors.
The length estimator is validated only as a Pearson correlation with ground-truth durations "comparable to other systems," with the numbers relegated to an appendix; for atypical speech the only safeguard is the clamp range.
The ethics section is candid: transcript-free cloning lowers the barrier to cloning a voice from arbitrary recordings, and the authors recommend consent requirements, watermarking, and synthetic-speech detection for any deployment.