Rethinking Automatic Music Mixing as Sequential Stem Blending
Yen-Tung Yeh, Chung-Jui Chan, Yun-Ning, Hung, Yi-Hsuan Yang
Amy
eess.AS
2026-08-06
Reformulates music mixing as blending one stem at a time into a growing submix via latent flow matching; tops 5 of 6 blind listening tests and crushes parallel systems on stem blending
Every existing automatic music mixing (AMM) system is parallelized: all input tracks go into the model together and the finished mix comes out in one pass. That carries two concrete costs. It cannot perform one of the most common operations in mixing engineering, integrating a single stem into an existing submix, which is the situation whenever a guitar is overdubbed or a vocal is re-recorded. And the explicit effects chain used by mainstream systems, a fixed topology of gain, EQ, compression, panning, and reverb, caps expressive power: any processing outside the predefined chain is simply not expressible. The paper asks the opposite question: human mix engineers work one track at a time, so why not the machine?
The reformulation is s(k) = s(k−1) + f(xk, s(k−1), Ck): at each step one stem is blended into the current submix, the submix serves only as context and is never modified, and the model outputs the processed stem that gets added back. Ck is a meta-conditioning set holding the integrated loudness of both audio, genre, and instrument type.
The transformation f is learned with latent flow matching. In the latent space of the Stable Audio Open VAE, a rectified flow transports the raw stem's latent to the processed stem's latent, conditioned on the submix latent. The design choice that matters: xk and yk share musical content and differ only in sonic treatment, so the model learns a content-preserving sonic transformation rather than generation from scratch, a much lighter task. The backbone is 2 MMDiT dual-stream blocks plus 2 DiT single-stream layers, with the submix latent injected as a second modality through bidirectional joint attention.
Training data is the hard part: real multitrack sessions never record the intermediate submix at each step. The solution simulates only the final blending step. MedleyDB v1 ships raw/wet stem pairs that form triplets directly; MoisesDB has only wet stems, so the raw side is manufactured with degradations: five parametric EQ modes, each the inverse of a specific mixing decision (masking boost lifts the frequencies the submix already occupies, over-cut notches the stem's own prominent bands, low-end mud, harshness adds a narrow 2–5 kHz peak, blend combines the first two), plus room reverb from simulated impulse responses. The degradations are engineered around the mistakes a mix engineer would actually correct, not random effect stacking.
Inference starts from a zero latent and processes stems one at a time, either in random order or a domain-knowledge order (drums → bass → guitar → keys → strings → vocals → other) that establishes rhythmic and tonal foundations first.
The training bill is surprisingly small: batch 128, 10-second segments at 44.1 kHz, 30 epochs on a single RTX 4090, 10 inference steps.
On the stem blending benchmark (300 segments from 24 held-out MoisesDB songs), the distributional distances separate cleanly:
| System | FxEnc++ KAD ↓ | CLAP KAD ↓ | RMS FD ↓ |
| Proposed (sequential) | -0.01 | -0.07 | 5.37 |
| Raw-mix (no processing) | 1.42 | 1.61 | 486 |
| MEGAMI† (parallel + reblend) | 7.06 | 7.72 | 2.2e+03 |
| DMC (parallel) | 46.45 | 1.76 | 1.71e+03 |
The most striking row is Raw-mix: doing nothing beats both dedicated mixing systems on FxEnc++ KAD (1.42 against DMC's 46.45 and MEGAMI's 11.67). Parallel models have no mechanism to recognize input that is already well mixed, so they modify the submix as well and make it worse. The sequential model's KAD sits near zero, its output distribution almost indistinguishable from the reference.
On full AMM (MedleyDB v2) the picture is mixed: Proposed-Domain reaches CLAP KAD 6.52 against MEGAMI's 10.11, and RMS FD 37.1 against 106, but loses FxEnc++ KAD 31.01 to 7.99 and trails on tonal balance. Domain-knowledge ordering consistently beats random.
Perceptual results are the strongest evidence. In a MUSHRA-style blind test with 18 musically trained listeners, the proposed model took the highest median score in 5 of 6 examples; its Meta Audiobox production-quality score of 7.971 leads Raw-mix (7.737), DMC (7.734), and MEGAMI (7.653). The test stems were regenerated by Moises's generation model, outside the training distribution.
Making blending a first-class operation lets interactive workflows fall out naturally: a user can supply their own submix at any point, inspect intermediate outputs, or remix a single track. Stem ordering, treated as a nuisance variable in prior work, becomes a training-free knob for mixing style. For audio tool teams this is the product-shape difference between one-click output and an interruptible process, and it shows a small model with the right task reformulation winning listening tests on a single GPU.
The authors concede that full-AMM performance on tonal balance and mixing-style similarity does not yet match state of the art, partly because training simulates only the final blending step and the model never sees sparse early submixes. Two evaluation caveats deserve weight: the stem blending benchmark is built with the same degradation strategy as training, effectively an in-distribution exam, leaving the 18-listener, 6-example perceptual test as the only out-of-distribution evidence; and the production-quality score comes from a model trained on 16 kHz mono audio, flagged in the paper's own footnote as a proxy. Objective metrics and perception diverge plainly (worse crest factor and tonal balance, best listening scores); the paper attributes this to listener taste, plausibly but without verification.