ByteDance SwanTale unifies multi-speaker voice, sound, and song in one 2B model
SwanTale: Unified Multi-Speaker Speech and Audio Generation for Instruct and Zero-Shot Tasks
Yu Zhang, Ruiqi Li, Changhao Pan, Ke Lei, Xiang Yin, Cheng Yang
eess.AS, cs.SD
2026-08-03
SwanTale (ByteDance) uses one 2B model to generate multi-speaker speech, sound, song, and music from a caption or a reference clip, hitting 0.95 zero-shot timbre consistency.
What problem this solves
In animation dubbing, audio drama, games, and short-video work, creators routinely design sound from nothing. Sometimes they describe the voice and the ambient scene in words; sometimes they hand the model a reference clip and ask it to copy that timbre. These two jobs have lived in separate systems. Instruct TTS produces speech with no environment, zero-shot voice cloning offers no natural-language control over style, and most pipelines emit speech only, leaving song, music, and sound effects to downstream mixers that drift in timing, loudness, and reverb. SwanTale, from ByteDance, tries to fold all of it into one model.
Method
SwanTale is an end-to-end speech and audio generation system with several pieces.
- SwanVAE (audio tokenizer). It compresses 48 kHz mono audio into 96-dimensional latents at 25 Hz (40 ms per frame), with an encoder receptive field of about 0.95 s and an end-to-end field of about 3.23 s; it has 407M parameters. Reconstruction uses multi-resolution STFT loss, multi-band Mel loss, frame-energy loss, KL regularization, and MPD/MRD/MBCSD discriminators. It is evaluated on speech, singing voice, general audio, and music.
- Data (SwanData-Caption). About 70M captioned records. A caption is not "male voice reading" but three layers: environment (location, sound field, reverb, persistent background), speakers (gender, age, persona, timbre, loudness, rate, accent, habitual style), and content (a timeline with <S1> speaker tags and <Audio> effect tags). Annotation uses Seed2.0 Lite with a style-persona library built for animation, short drama, and advertising.
- Flow-matching DiT. A Qwen-family text encoder injects the caption into every DiT layer through cross-attention; text uses the CosyVoice 2.0 tokenizer. Quality scores (STOI, SI-SDR, PESQ, MOS) become a caption flag, and inference forces "high".
- Unified MoE. Every other DiT FFN is replaced by a dynamic Top-PP MoE: audio experts are routed per frame, task-shared experts per sample, and a null expert acts as a fallback. The key device is a time-aware expert budget: the closer to the generation end, the more experts fire, because fine waveform detail needs more compute; capacity and null-expert bias also shift with time. SwanTale has about 2B active parameters.
- Curriculum. Four stages: a zero-shot base on 23M hours single-speaker plus 1.7M hours two-speaker, then dense caption adaptation, then a full caption mixture with Unified MoE (10M samples), then high-quality high-expressiveness SFT (1M samples).
- GRPO post-training. A marginal-preserving SDE serves as the stochastic flow policy for group-relative optimization with K=8 trajectories. Rewards split into shared terms (phone core, phone length, pause punctuation, edge RMS, quality) and task-specific terms (attribute agreement for instruct, speaker similarity for zero-shot). A KL constraint plus anchor replay prevents collapse. Inference uses two-stage decomposed CFG with text and full-condition guidance weights in [1.5, 3.0].
Results
On reconstruction, SwanVAE reaches PESQ 4.1683, STOI 0.9680, MCD 0.9638, and ViSQOL 4.1248 for speech, the best PESQ and MCD among all baselines; for singing voice it reaches PESQ 3.9821 and STOI 0.9001. On zero-shot monologue it scores Timbre Consistency 0.95, SpeechJudge 4.01, Content Error 0.046, and Expressive Richness 3.50, first on every metric against all compared systems. Benchmarks are the in-house SwanBench-Speech (monologue and dialogue), SwanBench-Scene (180 instructions, four dimensions scored 1 to 5), SwanBench-Caption (64 cases), and the public InstructTTSEval.
Why it matters
For anyone building speech and audio generation, unifying instruct with zero-shot and emitting speech, effects, song, and music from one model is an engineering simplification: no downstream pipeline to stitch, no reverb and timing drift to chase, because the model resolves it internally. The three-layer caption format (environment, speaker, content) is also a transferable answer to "how do you describe a piece of audio". The cost is weight: 2B active parameters, 64 A100 GPUs to train, and heavy annotation, so it is realistic for large labs to reproduce rather than individuals to follow.
Limitations
The paper has no dedicated Limitations section. From the text, general-audio and music reconstruction is not uniformly best (music is second on ViSQOL, general audio third on LSD), so non-speech modalities still trail. GRPO post-training targets single-speaker cases; multi-speaker and audio capability survive through anchor replay rather than direct reward. The 25 Hz latent frame rate (40 ms per frame) leaves fine waveform detail to the decoder. Unified MoE capacity needs careful tuning, with a z-loss and a null-collapse penalty supplied. Much of the evaluation rests on in-house SwanBench sets, so fairness against public systems is something to weigh.
Terms
- instruct / zero-shot: instruct generates from a text description; zero-shot clones the timbre of a reference clip.
- PESQ / STOI / MCD: objective speech-quality metrics. PESQ measures perceptual quality (about 4.5 max), STOI intelligibility (0 to 1), MCD Mel-cepstral distance (lower is better).
- flow matching: trains a continuous generative model by learning the vector field that flows noise to data; diffusion-like but with more direct sampling.
- CFG (classifier-free guidance): at inference, amplifies the conditional signal over the unconditional one to improve instruction adherence.
- MoE (Mixture of Experts): splits a large network into expert subnets and activates only a subset per step, trading compute for capacity.
Source
Related papers
All paper explainers