Dev Trains the Missing Encoder for YuE2, Enabling Custom Music Input

bdsqlsz · x · 2026-09-15

Developer sinceriously released yue2-mothersuperior-realaudio-tokenizer-v4 on Hugging Face, filling in the audio-to-semantic-token encoder that the open-source music model YuE2-3B never shipped, plus a NAR-branch LoRA so the decoder renders real-production latents.

tokenizerheadjointv4.pt: maps MERT-v2-FullSong layer-20 features (per-track instance-normalised, 25 Hz) to 32,768 YuE2 semantic codes; 8-layer transformer, d=512, 512-frame windows. Held-out top-1 exact match on YuE2's own songs is 16.1%, though near-miss codes render almost identically and NAR round-trip ear tests sit around 95%.
narlorajointv4.pt: rank-32 LoRA on narselfattn.{q,k,v,o}proj and narmlp.{gate,up,down}proj across 28 layers, plus full vae2llm/llm2vae, trained jointly with the head on real audio.
Training: 4,765 YuE2 self-generated songs first, then adapted to real audio; a 100 MB minted regularizer pack (4,732 generated songs) is required, with 50% of AR training songs drawn from it to prevent small artist sets from collapsing YuE2's token grammar.
Requirements: Python 3.12 with yue2-infer, torch 2.10/cu128, demucs, etc.; a 24 GB GPU suffices for every stage (14–18 GB measured).

Use case: tokenize your own recordings, LoRA-tune YuE2's AR branch on an artist, and generate new songs or covers.

Original post →

Dev Trains the Missing Encoder for YuE2, Enabling Custom Music Input

More from Multimodal