YuE2: Unifying Symbolic and Audio Music Generation at Frontier Quality
Ruibin Yuan, Jiahao Pan, Junyan Jiang, Zhiyue Wu, Ziya Zhou, Jiankai Sun, Yizhi Li, Ge Zhang, Yicheng Gu, Zeyue Tian, Junyu Dai, Hanfeng Lin, Kai Li, Shangda Wu, Xuanjie Liu, Jiaming Wang, Zihan Liu, Yue Wang, Yinghao Ma, Hanzhi Yin, Kangrui Chen, Xinyue Zhang, Ziyang Ma, Mengqi Liao, Hejia Zhao, Guowei Huang, Chao Yan, Lei Ke, Jianwei Yu, Bei Liu, Joe Guo, Liumeng Xue, Gus Xia, Wei Xue, Yike Guo
eess.AS, cs.LG, cs.SD
2026-09-28
HKUST's 3.58B YuE2 writes an ABC score then a 48 kHz song. SongBench Avg is 6.73 on WildSongBench (6.96 best-of-8). Same-checkpoint experts prefer planning 49.3% to 34.6%.
Symbolic music models make melody, harmony, rhythm, and form inspectable, then stop before a finished recording. Audio song models emit a complete track and leave those decisions buried, so changing a hook or a chord progression is mostly prompt luck.
The two families rarely share a checkpoint. Ordinary recordings also lack aligned notes, chords, beats, and sections, which is why a score-then-audio generator is hard to train at scale. YuE2 asks whether one model can write a readable score, perform it, and still land at frontier full-song quality.
Generation is framed as progressive musical commitment: each stage settles some decisions and leaves the rest below. The picture is a lead sheet written before a tracking session. The microphone no longer decides the hook. Text and lyrics state intent. An ABC score locks melody, chords, key, meter, tempo, and form. A 25 Hz semantic token stream fills in performance detail that notation skips. A 25 Hz, 64-d acoustic latent sequence then feeds a separately trained 48 kHz stereo VAE.
A 28-layer AR-NAR Mixture-of-Transformers (about 3.58B parameters) runs both stages. The autoregressive stream predicts the score then the semantic tokens, left to right. Once those sequences exist, the non-autoregressive stream predicts flow velocity over all acoustic frames with bidirectional context. The streams share attention and keep separate norms, projections, and MLPs. Discrete positions cannot read the acoustic target; acoustic positions see the full text, score, and semantic plan. The loss is 0.25 times next-token cross-entropy plus conditional flow matching.
Training uses about 346,000 hours of music, packing whole songs into a 24,576-position context. Four input mixtures include or drop the score and the semantic tokens, so the same checkpoint can generate with or without planning.
Supervision is built offline from recordings. SheetSage2 recovers a lead sheet. A MERT2 tokenizer emits one ID every 40 ms from a 32,768-entry codebook (375 bit/s). Those codes are fused from frozen MuQ and Qwen2-Audio-Instruct views; a new encoder is then trained to predict them. SheetSage2-AR trains only on labels from a non-autoregressive Prober.
Creation, editing, and covering share the score interface. The model writes the score, consumes an edited score as a frozen prefix, or renders a SheetSage2 transcription under a new style. No original-cover pairs are used.
WildSongBench has 192 in-the-wild prompts, 94 labeled Chinese and 98 English. The standard protocol keeps the lower-PER song of two candidates.
| System | SongBench Avg | AudioBox PQ | Note |
| YuE2 | 6.73 | 8.26 | Best public SongBench Avg |
| YuE2 best-of-8 | 6.96 | 8.27 | Highest observed SongBench Avg |
| Mureka 9 | 6.94 | 8.02 | Best proprietary SongBench Avg |
| Suno v5 | 6.87 | 8.17 | Strongest text alignment in the full set |
| LeVo 2 | 6.32 | 8.40 | Highest public PQ |
| HeartMuLa | 6.25 | 8.29 | SongEval Avg 4.55 vs YuE2 4.26 |
Best-of-8 ranks eight samples by SongBench Musicality, then Q3O, then PER. AudioBox PQ is not used for selection; both YuE2 rows beat every proprietary system on PQ. Lyric PER is 0.0844; MiniMax Music 3 is lower at 0.0627. HeartMuLa's SongEval lead is flagged in the paper as using SongEval and AudioBox in post-training.
Expert listening keeps 4,439 pairs. Best-of-8 versus Suno v4.5 is 57.3% to 30.5% on overall quality; versus Suno v5, 40.4% to 39.9%; versus Suno v6, 31.6% to 59.3%. Tie-adjusted audio-quality preference averages 58.9% across six proprietary systems.
Same-checkpoint planning wins overall quality 49.3% to 34.6% (p=0.0070) and musicality 45.0% to 29.4%. Unified MoT versus a matched 3.4B LM+DiT: overall 53.4% to 35.6%, audio quality 48.5% to 29.6%; text alignment is 34.0% versus 33.5%.
Rendered audio matches its own score at 0.9464 melody and 0.9246 chord similarity, versus 0.2160 and 0.2473 for a mismatched score under the same prompt. Local score edits hit 84.17% target pitches and 79.54% target chords, while unedited melody and harmony stay between 90.16% and 94.34%. Zero-shot covers of 948 unseen SHS100K works reach CLEWS mAP 0.647 with a full score, against 0.419 for SongEcho; dropping the score collapses mAP to 0.006.
MERT2-30s beats prior numbers on 14 of 15 MARBLE metrics. SheetSage2-AR leads 12 of 15 transcription pairs and beats its own label generator on 10 of 15.
For people building song models, compose-then-perform is now an ablatable mechanism. A matched checkpoint shows that writing the score raises expert preference, and that a unified MoT beats a split LM+DiT with the same representations. Readable ABC also gives an external language model something to edit, so an agent can revise notes rather than only the prompt. The 3.58B checkpoint is public and smaller than six of eight public baselines.
The 6.96 best-of-8 mean is not a single-sample win over Suno v6. Under the two-candidate protocol, 6.73 still sits below Mureka 9 at 6.94 and Suno v5 at 6.87, and listeners clearly prefer v6. This is an open system in the proprietary band, not a sweep.
The paper has no standalone limitations section. Several numbers already qualify the headline.
Best-of-8 selects on SongBench Musicality, then reports the highest SongBench mean in the comparison. That loop is only partly closed by AudioBox PQ, which was not used for picking. Eight candidates are also a larger budget than the two-song proprietary interface the standard protocol tries to match.
Human and automatic rankings diverge. Bo8 beats Suno v6 on SongBench Avg, yet overall expert preference is 31.6% to 59.3%. Audio quality is the clear strength; the whole-song vote is not.
Score-audio metrics run a frozen SheetSage2 transcriber, later expert-corrected, on a model whose symbolic targets came from the same family. Residual family bias is possible. Local edits still regenerate the full song. Agentic editing is one case study (The Last Train, nine curated steps, fourteen renders) with no controlled comparison. Lyric PER is not the public best, and Bo8 raises it from 0.0844 to 0.0979. In covers, dropping chords improves style alignment and Musicality while hurting work-identity retrieval.