StepAudio 3 Music Technical Report
Chengli Feng, Zhiyue Wu, Jiahao Song, Zheqi Dai, Boyang Wang, Ruibin Yuan, Junming Gong, Wenxiao Zhao, Jing Guo, Gang Yu, Xiangyu Zhang, Xuerui Yang, Chao Yan
eess.AS, cs.SD
2026-09-11
StepAudio 3 Music uses a 50 Hz single codebook and a 0.9B DiT to 48 kHz. It leads AudioBox CE/CU/PQ and MuQ-MuLan 0.4465; arena Elo 1105, behind Suno V5.5 and Mureka.
Thirty seconds of pleasant audio is no longer the hard part of music generation. The hard part is keeping melody, section changes, lyrics, and arrangement aligned over several minutes, and letting a creator edit "the chord in this bar" instead of only a mood prompt. Diffusion and flow-matching already render timbre well; language models own long-range structure. The plan in the middle is usually opaque: quantized CLAP in MusiCoT, melody tokens in Qwen-Music, metadata in ACE-Step 1.5. If a musician cannot read it, they cannot revise it.
StepAudio 3 Music, from StepFun with ACE, CUHK, and UC San Diego, writes that plan in ABC notation (ABC-CoT), predicts a 50 Hz single-codebook token stream, and renders 48 kHz audio with a flow-matching DiT. The training mix is more than 100 million songs, 5 million hours, and about one trillion tokens. Maximum length is 5 minutes 30 seconds.
Three parts: a tokenizer, a MoE autoregressive model, and a DiT renderer. The codec is trained and frozen; the LLM only predicts tokens.
The discrete bottleneck is the real design fight. Single-codebook VQ, Semantic RVQ, and Acoustic RVQ were compared at a shared 25 Hz. Multi-codebook RVQ reconstructs better from ground-truth tokens. Once tokens are predicted, Acoustic RVQ sounded poor internally, Semantic RVQ kept fidelity but lost musicality as errors piled across residual books, and single-codebook VQ gave up a little fidelity to keep musical development stable. After locking the single book, raising the rate from 25 Hz to 50 Hz left fidelity similar and improved musicality, doubling supervision from the same corpus. The final tokenizer is a 0.6B bidirectional Conformer with 65,536 entries, one token per frame, nominally 800 bit/s.
Renderer scaling goes the other way. Holding the tokenizer and ground-truth tokens fixed, a 0.9B DiT beats 4B and 8B on MCD, Mel/STFT L1, SI-SNR, and UTMOS. Extra renderer capacity cannot recover information the tokens never stored. Long audio is rendered in 30-second chunks, each seeing the last two seconds of VAE latents from the previous chunk.
ABC-CoT is a two-pass factorization. Pass one writes tempo, meter, key, chords, bars, and melody from lyrics, a text prompt, and optional references. Pass two predicts music tokens conditioned on that score. Music-to-ABC and ABC-to-music are both trained.
The data pipeline filters bandwidth, detects events, separates vocals with Demucs, IDs language with MMS-LID, transcribes Chinese and English with FireRed ASR and other languages with WhisperX, then sections songs with SongFormer. The LLM starts from a text MoE and consumes about 600B + 500B + 80B tokens across pretraining, midtraining, and annealing, with ABC-CoT about 16% of the mid mix. SFT is about 20B tokens covering lyrics-to-song, instrumental, dry-vocal accompaniment, and covers. DPO uses about 500 prompts, four caption-lyrics instances each, four candidates per instance, and the widest expert preference gap in each group, about 2,000 pairs.
The main objective table uses 339 shared lyrics-to-song items for AudioBox and MuQ-MuLan, and 316 vocal items for SongBench, against Suno V5.5, Suno V5, Mureka V9, and MiniMax Music 3.
| Metric | StepAudio 3 Music | Strongest baseline |
| MuQ-MuLan | 0.4465 | Suno V5.5 0.4437 |
| AudioBox CE | 7.7086 | Suno V5 7.6823 |
| AudioBox CU | 8.0052 | Suno V5.5 7.9584 |
| AudioBox PQ | 8.3868 | Suno V5.5 8.2988 |
| SongBench Musicality | 5.7465 | Mureka V9 5.9382 |
AudioBox's three ranked axes and caption similarity go to StepAudio. Mureka V9 leads the SongBench dimensions. On the preliminary Artificial Analysis Music Arena Vocals board, Quality Elo is 1105, fourth in the snapshot, behind Suno V5.5 and Mureka and ahead of Suno V5 and MiniMax. The arena asks each system to write its own lyrics, so it is not a rerun of the internal fixed-lyrics set.
DPO versus SFT: Musicality 5.6395 to 5.7465, SongBench seven-dimension mean 6.5445 to 6.6438, MuQ-MuLan 0.4157 to 0.4465. Adding ABC-CoT at inference moves the mean from 6.6438 to 6.6713. One round of text-LLM plan editing reaches 6.7220. Mixing, arrangement, and melody move most; vocals barely move. Self-written scores help a little. Edited scores help more.
A codec built for generation is not a codec built for reconstruction. More RVQ layers and better SI-SNR do not mean an autoregressive model can finish a song. A 0.9B DiT beating an 8B renderer says the same: the bottleneck is the token, not the painter. ABC as an editable mid-layer is closer to an arranging workflow than a hidden CoT. The score lift is small. Editing the score is worth more than asking the music model to write its own theory.
This is a closed-system report in Suno and Mureka's neighborhood. Five-and-a-half-minute songs, covers, and dry-vocal mixes are in the task list. The headline table only proves lyrics-to-song.
The authors are blunt. The reported subsets are mostly vocal and lyrics-conditioned; instrumental, cover, and vocal-to-mix lack head-to-head tables. Caption similarity is not note-level ABC adherence. SFT ranked data with SongBench Musicality, so that score is not independent of selection. One sample per condition, no seed variance. Baselines came from hosted product APIs. ABC-CoT plus LLM editing is a full inference recipe, not a factorial ablation.
Elo 1105 is a preliminary board; the paper does not print a full confidence-interval table. Complex arrangements, very long dependencies, vocal naturalness, and high-frequency reconstruction are listed as open. Internal expert listening supported the single-codebook choice; that round was not a separate controlled listening study.