Separate-and-Detect: Unified Drum Transcription and Stem Generation via Latent Diffusion
Wei-Han Hsu, Chih-Cheng Chang, Bo-Yu Chen, Li Su, Yi-Hsuan Yang
cs.SD
2026-08-02
Drum transcription as separate-then-detect: a five-stem latent diffusion separator beats a U-Net baseline in overall F1 (p<0.01) and end-to-end ADT on kick and snare, plus editable stems.
Automatic Drum Transcription (ADT) turns a full music mix into symbolic drum events: which beat is a kick, snare, tom, hi-hat, or cymbal hit. The hard part is that in a full mix the drums sit under vocals, harmonic instruments, and other sources. Mainstream systems predict drum events end-to-end straight from the mix; that is efficient, but it only emits symbols and cannot recover the isolated, editable waveform (the stem) of each drum.
Music production needs both. Producers do not just transcribe drums, they replace, layer, and remix individual drum tracks, especially kick and snare, which carry the groove. This ISMIR 2026 paper revives an older idea, separate then detect: first split the drums into five editable stems, then run a fixed onset detector on each stem to get symbolic events. The intermediate representation becomes explicit, and symbols and stems come out together.
The pipeline has three stages. First, a frozen HT-Demucs (a general music source separation model) pulls a drum-dominant signal out of the full mix. Second, the core: a five-stem latent diffusion separator that jointly generates kick, snare, toms, hi-hats, and cymbals in a compact VAE latent space, conditioned on the mixture latent plus five learnable track tokens; after denoising, a VAE decoder produces mel spectrograms and a HiFi-GAN vocoder renders waveforms. Third, a frozen madmom CNN onset detector runs on each rendered stem, with peak-picking parameters tuned once on a development split via Optuna and then frozen.
The separator also carries two auxiliary branches that act only during training and are discarded at inference (under 0.06% of trainable parameters combined). An onset branch (OB) predicts coarse per-stem onset maps from decoder features, forcing the model to keep the impulsive energy envelopes drums need. A timbre branch (TB) uses an MLP to predict 7-D DrumGAN descriptors (brightness, hardness, depth, roughness, boominess, warmth, sharpness). Total loss is the diffusion noise-prediction loss plus two weighted auxiliary losses.
Training uses synthetic StemGMD (1200+ hours) plus a small real-acoustic set, IDMT-SMT-Drums (2.1 hours), with evaluation on two unseen real datasets, MDB Drums and ENST-Drums.
On separation quality, every MSG-LD variant clearly beats the U-Net baseline LarsNet in mel-spectrogram MSE and Frechet Audio Distance (FAD). LarsNet's mean FAD is 1.42; the MSG-LD variants come in far lower, and the OB+TB variant reaches the lowest FAD on kick (0.23), snare (0.09), and toms (0.13).
Transcription is the point. With the frozen detection backend, the MSG-LD+OB pipeline consistently beats LarsNet in overall F1: 0.707 vs 0.613 on MDB and 0.640 vs 0.493 on ENST (Wilcoxon p<0.01). Against the end-to-end system ADTOF, MSG-LD+OB pulls ahead on kick and snare: MDB kick 0.931 vs 0.851, snare 0.760 vs 0.752.
| System | MDB overall F1 | ENST overall F1 | MDB kick F1 |
| ADTOF (end-to-end) | 0.795 | 0.709 | 0.851 |
| LarsNet (U-Net) | 0.613 | 0.493 | 0.899 |
| MSG-LD+OB (this work) | 0.707 | 0.640 | 0.931 |
The ablation shows +OB is the most balanced configuration for transcription, with the highest mean overall F1 across the two datasets (0.674); the timbre branch trades some spectral reconstruction accuracy for distributional quality. ADTOF still has the highest overall F1, because it stays stronger on dense hi-hat and cymbal textures where separation-based methods still struggle.
For audio, music information retrieval (MIR), and music-tool builders, the draw is not another leaderboard transcription model. It is that generative separation can serve as a transcription front end, and hand back the editable stems that direct-prediction systems cannot produce. Kick and snare carry the groove, and beating end-to-end ADT on those two means the separation route already holds its ground on the drum pieces that matter most.
The authors list these plainly. The 16 kHz sample rate saves compute but discards high-frequency content, hurting hi-hats and cymbals. Diffusion sampling is slow: about 25 seconds to generate a 10-second sample on an RTX 6000 Ada, far slower than a single forward pass in ADTOF. Dense broadband textures (cymbals, hi-hats) remain a bottleneck, with inter-class confusion between hi-hats and cymbals, and VAE latent compression is a further limit.
One more caveat: training leans heavily on synthetic StemGMD, with only 2.1 hours of real acoustic data, so generalization to real recordings is not deeply stressed. The paper also skips waveform-domain separation metrics like SDR or SI-SDR (arguing they entangle with vocoder reconstruction and phase), so the absolute separation quality lacks a standard yardstick for comparison.