RSE-TTS reuses TTS backbone as speaker encoder, cutting params 48%

kastnerkyle · x · 2026-11-13

A new arXiv paper introduces RSE-TTS, showing a pretrained autoregressive TTS backbone can be repurposed as a reference speech encoder, compressing paired reference audio and transcript into 8 continuous vectors to boost speaker similarity.

Sharing the generation backbone cuts encoder-generator parameters by 48% vs independent pretrained encoding
The 118M-parameter generator beats official MOSS-TTS-Nano cloning modes on WER and speaker similarity across all three English benchmarks
With acoustic continuation it approaches much larger systems' similarity while achieving lower WER than CosyVoice 3 and MOSS-TTS-Local
Ablations and activation interventions trace how text benefits flow through intermediate acoustic representations

Original post →

RSE-TTS reuses TTS backbone as speaker encoder, cutting params 48%

More from Multimodal