RSE-TTS: Compact Text-to-Speech through Shared Text-Aware Reference Encoding and Generation
Ming Ma, Mingjie Chen, Yuanzhong Zheng, Hengyao Bao, Hailin Yu, Shijing Yu, Hao Qiu, Gongfan Cao, Xiao Li, Xuemin Zhao, Youhai Jiang
cs.SD, eess.AS
2026-11-11
Sharing one pretrained TTS backbone between encoding and generation cuts params 48%; the 118M RSE-TTS beats official MOSS-TTS-Nano on WER and similarity across three benchmarks.
Zero-shot voice cloning asks a TTS system to read unseen text in the voice of a short reference recording. Two approaches dominate. Acoustic continuation prepends the reference to the generation history, prompt-style; it preserves the most voice detail, but the reference's linguistic content, recording artifacts, and atypical pauses ride along, and longer references consume generation context. A reference encoder instead compresses the recording into a fixed-size condition. The interface is clean, the hard part is keeping speaker identity and expressive detail through compression. Compact models feel this tension most, since their parameter budget cannot hold a second large transformer. The question here is direct: can a pretrained TTS generator serve as the voice encoder?
The base is MOSS-TTS-Nano: a Global transformer (12 layers, hidden 768) models the text and acoustic-frame sequence, a single-layer Local transformer generates codec tokens within each frame, and a frozen codec represents speech at 12.5 frames per second with 16 codebooks.
RSE encoding is a single forward pass through the shared backbone. The input concatenates reference text embeddings, reference speech frame embeddings, an EOS marker, and eight learned queries. Under a causal mask the queries attend only to the preceding reference; their outputs pass through layer normalization and a learned linear bridge, giving eight compressed vectors, to which a projected 192-dimensional CAM++ speaker embedding from a frozen extractor is added.
Generation runs through the same Global transformer, starting from the compressed condition, then the target text, then frame-by-frame synthesis. Training uses one loss only, the autoregressive cross-entropy of target speech under teacher forcing; gradients flow from target predictions back through the eight vectors, so encoding and generation learn together. Training pairs are two different utterances from the same speaker (with IPA augmentation on transcripts), which forces the vectors to carry identity across a change of content.
Each design choice is backed by an ablation. Pretrained initialization transfers the backbone's text-speech alignment to compression, lifting Seed-TTS SIM from 0.400 to 0.530. The paired transcript gives the encoder both what was said and how it sounded. Sharing removes the second transformer, cutting combined parameters from 228.5M to 118.2M, at a small similarity cost (0.552 vs 0.564 on Seed-TTS).
Five variants, one epoch on the same 20k-hour subset:
| Variant | Params | Seed-TTS WER/SIM | CV3-eval WER/SIM | MiniMax WER/SIM |
| Independent, random init | 228.5M | 1.39 / 0.400 | 2.44 / 0.342 | 0.75 / 0.391 |
| Independent, pretrained init | 228.5M | 1.46 / 0.530 | 2.63 / 0.469 | 0.91 / 0.592 |
| Independent, text + audio | 228.5M | 1.41 / 0.564 | 2.50 / 0.507 | 0.75 / 0.651 |
| Shared, audio only | 118.2M | 1.35 / 0.498 | 2.46 / 0.435 | 0.80 / 0.553 |
| Shared, text + audio (RSE) | 118.2M | 1.33 / 0.552 | 2.63 / 0.499 | 0.64 / 0.615 |
The eight vectors hold more than expected. Growing the reference from 3 to 24 seconds raises SIM from 0.378 to 0.569, with all 30 EARS speakers improving at 24 seconds, while generation always receives the same eight vectors. They also carry delivery: with the speaker embedding held fixed, fast references shorten output by 7.0%, slow references lengthen it by 13.4%, and high-pitched references raise pitch by 6.01 semitones on average. And they complement the conventional embedding: a linear probe predicting the WavLM embedding of another recording by the same speaker scores about 0.44 from either representation alone and 0.469 from both, improving on 58 of 63 held-out speakers.
The mechanism experiments decompose why text helps. Reversing the reference transcript's token order leaves the token inventory intact but breaks coherence and text-audio correspondence; SIM drops from 0.538 to 0.497. Activation patching then swaps hidden states between the correct-text and reversed-text runs: restoring reference-audio states at layers 3 and 6 recovers SIM to 0.545, and restoring the eight query states at layer 9 recovers it to 0.542. The text benefit first lands in intermediate acoustic representations, then consolidates into the compressed vectors.
After full-scale English training, the 118M generator beats official MOSS-TTS-Nano's corresponding cloning modes on both WER and SIM across all three benchmarks. Adding acoustic continuation on top, it posts lower WER than the evaluated CosyVoice 3 and MOSS-TTS-Local on each benchmark while approaching their similarity, with a 129M generation chain that is 6.6x and 30.5x smaller respectively. A continuation-only model trained from the same initialization and data does reach higher similarity (1.45/0.591 vs 1.33/0.552 on Seed-TTS); RSE buys lower WER and a fixed-size reference interface.
For on-device and compact cloning TTS this is a copyable architecture decision: one backbone does encoding and generation, saving 48% of the combined parameters. The transfer result, that pretrained generator weights work as encoder initialization, plus the causal patching analysis, moves reference conditioning from black-box tuning toward mechanism. To be honest about the magnitude, the quality gain over MOSS-TTS-Nano is incremental; the contribution is efficiency and analysis, not a new quality ceiling.
The authors list three: fixed-size conditioning still trails continuation in similarity, with the gap present from the start of generation, which they attribute possibly to teacher forcing encouraging reliance on acoustic history, with compression loss as another candidate cause; a reference transcript is required, so audio-only deployment needs ASR and its robustness is untested; experiments are English-only.
Close reading raises more. The training corpus is in-house and undisclosed, which hurts reproducibility. Compared systems run their official default sampling settings, so sampling effects on the metrics are uncontrolled. The main text gives no head-to-head numbers for pure RSE against CosyVoice 3 or MOSS-TTS-Local; the WER-win claim covers only the RSE-plus-continuation combination. And there is no human listening test, so naturalness rests on SIM and WER alone.