StepAudio 3 Gen Technical Report
Bin Lin, Bo Zhao, Boyang Wang, Boyang Zhang, Boyong Wu, Chao Yan, Chen Geng, Chen Wu, Cheng Yi, Chengli Feng, Chenglin Zhu, DanNi Wan, Daxin Jiang, Dongqing Pang, Fei Tian, Feng Tian, Future Li, Gang Yu, Guanglong Yang, Jia Peng, Jiahao Song, Jiamin Fan, Jiangjie Zhen, Jianzheng Gao, Jun Chen, Li Xie, Lifang Zhang, Lingli Ji, Liying Shi, Lun Cai, Min Xu, Na Wang, Peilin Li, Peng Yang, Pengfei Tan, Qingjian Lin, Ruijie Xiong, Runze Li, Shenghua Hu, Shi Qiu, Siqi Tu, Siyi Zhou, Tianjiao Deng, Wanying Lu, Weiming Niu, Wen Sun, WenWen Qu, Xiangyu Zhang, Xianwei Zhang, XiaoSu Su, Xing Chen, Xinyu Liu, Xuerui Yang, Yang Li, Yang Yang, Yechang Huang, Yibo Zhu, Yifan Zhang, Yiyang Xu, Yu Fu, Yu Luo, Yu Zhou, Yumang Wang, Yunzhou Ju, Yuxiang Yang, Zekai Liu, Zengwei Yao, Zhenwei Mou, Zheqi Dai, Zhiyue Wu, Zichao Zhou
cs.SD, eess.AS
2026-09-11
StepAudio 3 Gen models speech, vocals, music and sound on one 12.5 Hz RVQ stream, winning 82.0% of Chinese TTS pairwise tests and 85.2/77.7 on InstructTTSEval ZH/EN.
Speech, sound effects, music, and singing have been modeled on separate tracks. TTS systems chase intelligibility and speaker identity; text-to-audio models chase events; text-to-music models chase structure and timbre. Any product that needs dialogue, laughter, underscore, and effects on one timeline currently glues together incompatible representations and pipelines.
Unified generators mostly live in a continuous latent space and render with diffusion or flow matching. The other family turns audio into discrete units and treats generation as language modeling. Discrete units fit an LLM vocabulary, a causal objective, and interleaved text–audio context. High-fidelity residual vector quantization (RVQ) still hurts: flattening every codebook along time makes the sequence unusable, while keeping only the coarsest layer throws acoustic detail away. StepFun uses a time–depth split and then attacks two interference channels that appear when audio is bolted onto a pretrained text LLM: summed multi-codebook embeddings that do not match text embedding statistics, and a 15-layer residual loss that can drown the backbone gradient.
Two pieces. StepAudio Tokenizer encodes speech, music, and general audio at 12.5 Hz into a shared 16 × 2048 residual code space and reconstructs 24 kHz waveforms. Following X-Codec, a frozen self-supervised encoder supplies semantic features, a convolutional encoder reads the waveform, the two are fused and downsampled, and a single RVQ quantizes them jointly so every layer carries both kinds of information. Codebook lookup is factorized cosine similarity. The decoder is fully causal for streaming: a Vocos-style Transformer with RoPE and 25-frame sliding-window attention, then an ISTFT head, with no look-ahead.
Codebook 0 is promoted into the LLM vocabulary as 2,048 audio tokens and predicted autoregressively along time. At each audio frame, the backbone hidden state plus codebook 0 condition a four-layer causal Transformer that fills in the remaining 15 codes along the codebook axis. The full 16-code frame goes to the codec. Predictor outputs never feed back into the backbone, so the two halves are trained jointly only after the predictor has converged.
On the input side, 16 embedding tables are summed and passed through the RVQ Adaptor, a token-wise residual stack with zero-initialized down-projections, added only at audio positions. Text positions never see this path. Zero-init makes the adaptor an identity map at the start.
Pretraining has four stages and about 2.7T tokens. Stage 1 freezes the backbone and trains only audio embeddings and the adaptor on ASR-style alignment. Stage 2 unfreezes the model on a 1:1 mix of audio understanding and text, still with no audio targets. Stage 3 introduces generation with a 3:1:2 mix of text, TTS, and interleaved dialogue, keeping text at 50%; the residual predictor is detached from backbone hidden states so the 15-code acoustic loss trains the predictor alone. Stage 4 restores gradients, drops the residual loss weight λ from 1.0 to 0.1, and stretches context from 16,384 to 32,768, roughly 44 minutes of audio. From Stage 2 on, half of every optimizer step is text. The tokenizer is pretrained on about 700k hours, then the encoder is frozen while the decoder is refined.
Requests use three fields: ROLE for speaker identity and timbre, DIRECTOR for scene and intent, SCRIPT for timed dialogue plus sound or music cues. Post-training is full-parameter SFT on about 5,000 hours, then GRPO with 16 samples per instruction. An audio understanding model captions each sample; a text LLM scores instruction consistency 0–100; speech items multiply that score by a CER/WER penalty and zero the reward if recognition error exceeds 0.5.
The RVQ Adaptor lifts audio-language metrics after pretraining:
| Setting | AISHELL-1 CER | LibriSpeech WER | MMAU Acc. | SpeechMMLU T2S |
| w/o Adaptor | 5.25% | 6.00% | 40.70% | 12.13% |
| w/ Adaptor | 3.00% | 3.41% | 51.70% | 58.76% |
CoVoST En→Zh BLEU moves from 12.05 to 30.56, Zh→En from 5.99 to 18.59.
Interference-aware pretraining also keeps more text skill than a three-stage baseline: MMLU 69.20 vs 64.99, MATH 45.07 vs 36.62, HumanEval Pass@1 54.27 vs 47.56.
TTS is judged only on perceived human-likeness. Across 1,500 Chinese pairwise comparisons, StepAudio 3 Gen posts Elo 1755.33 and an 82.0% aggregate win rate against Qwen-Audio-3.0-TTS-Plus, Doubao, MiniMax-Speech-2.8-HD, Inworld-TTS-2, and StepAudio 2.5 TTS (73.0%–90.0% per opponent). CER and speaker similarity are omitted on purpose; both can look fine on flat, inexpressive speech.
Voice design uses the full InstructTTSEval set of 6,000 utterances, judged for style consistency by Gemini 3.1 Pro. Chinese average is 85.2 against 74.0 for Qwen3-TTS-12Hz-1.7B-VoiceDesign; English is 77.7 against 73.5. A 490-battle blind Elo study ranks it first at 1668.5 with a 75.5% aggregate win rate. Vocals, music, sound, and vibe speech are shown as capabilities, without a comparison table.
Three engineering choices are the usable part. Time–depth modeling, rather than MusicGen-style delay patterns, keeps the pretrained LLM off residual-code prediction. A zero-init residual adaptor beats attaching a separate sequence encoder. Detaching residual gradients during the main generation stage stops a 15-way summed acoustic loss from rewriting the backbone. Discrete autoregression can cover several audio domains without a diffusion vocoder. Reproducing the stack is another matter: 2.7T tokens and an in-house tokenizer. The TTS listening wins are real; music and sound still sit at demo quality.
The report has no Limitations section. TTS listening tests are Chinese-only, with one voice per opponent. Voice-design scores come from Gemini 3.1 Pro, so the judge can play favorites. Singing, instrumental pieces longer than 60 seconds, and mixed scenes have no quantitative baselines. Backbone parameter count is undisclosed. Because the residual predictor does not feed back, within-frame acoustics cannot help the next-frame plan. SFT music and some singing come from StepFun's own Music Model, a self-reinforcing loop. The GRPO reward chain (caption then text-LLM score) is close-kin to the generator.