SwanTale: Unified model generates speech, sound effects, singing and music in one waveform
Scobleizer · x · 2026-11-09
The second technical report in the Swan series introduces SwanTale, a unified multi-speaker speech and audio generation model supporting both instruction-based generation and zero-shot voice cloning.
Key points:
- Jointly generates speech, ambient sound, local sound effects, singing, and music within a single waveform
- Data side: SwanData-Caption pipeline cleans raw audio, adds targeted synthetic coverage, and annotates multi-level captions
- Model side: SwanVAE for high-quality multi-audio-modality generation, plus reward-conditioned quality control, Engram conditioning, and Unified MoE for multi-task modeling
- Achieves leading results on key zero-shot and instruct benchmarks, with strong expressiveness across tasks
Targeted at animation dubbing, audio dramas, games, podcasts, and short-video production. Paper: arXiv 2608.02023.
More from Multimodal
- Designer builds AI pipeline to generate highway interchanges with error flagging — tomjohndesign · 2026-11-09
- FoleyForge: instruction-based sound effect editing model hits SOTA instruction adherence — kastnerkyle · 2026-11-09
- Kling 4.0 prompt demo shows how to generate dynamic action scenes with camera movement — umesh_ai · 2026-11-09
- Video-Coder: first code-driven video generation agent trained with execution rewards — UCF-Ai-LLMs · 2026-11-09
- OmniLatent: unified latent for part-aware 3D asset generation from a single image — Yunhan Yang · 2026-11-09
- Midjourney sref 4015763981 Renders Cartoon IPs as Glossy Art Toys — azed_ai · 2026-11-09