StemFX predicts per-stem FX chains as tokens, running 4000x faster than iterative mixing transfer

StemFX: Learning Mixing Style Representations via Autoregressive FX Chain Prediction on Source-Separated Stems

Yuan-Chiao Cheng, Jui-Te Wu, Brian Chen, Yen-Tung Yeh, Yu-Hua Chen, Yi-Hsuan Yang

cs.SD, eess.AS

2026-07-17

StemFX predicts per-stem FX chains as tokens, trained on separated stems and 85 effects, hitting 86.8% Top-1 retrieval, 4000x faster than iterative optimization.

What problem this solves

A song's mixing style is the chain of decisions a mix engineer makes: level balancing, spatial placement, and the FX chains applied to each stem, meaning which effects, in what order, with which parameters. Those FX chains carry most of the style. Transferring one song's mixing style onto another has not worked well. Some methods model only the stereo mix and never see per-stem FX chains; others fix the number and type of effects per track; others need differentiable effect implementations or scarce multitrack datasets (MUSDB18 has 150 tracks). StemFX sets out to learn variable-length per-stem FX chains.

Method

StemFX frames FX chain prediction as sequence generation. Given original stems xorig and effect-laden target stems xaug (four stereo stems: vocals, bass, drums, other), it predicts the per-stem FX chain F that turns orig into aug, where each chain is an ordered list of effect-parameter pairs.

Three pieces do the work:

A BSFiLM encoder: each stem becomes a log-mel spectrogram, the frequency axis is split into overlapping sub-bands, and each sub-band runs through its own two-layer CNN. A FiLM module injects 64 hand-crafted mixing features (dynamics, spectral, stereo, inter-stem), and a learned attention pool over time yields a 512-dimensional embedding.
An FX chain vocabulary of 358 tokens: 85 effect names, 164 parameter names, 101 quantized value bins, plus special and stem tokens. A chain reads as "[BOS] vocals soxgain soxgaingaindb value55 ...".
An FX chain generator: orig and aug each pass through the shared BSFiLM encoder to give two conditioning vectors, fed to a 6-layer Transformer decoder that predicts tokens autoregressively, trained with teacher forcing and cross-entropy. The whole model is 28.0M parameters.

The data pipeline is the other highlight. Genuinely labeled multitrack FX data barely exists, so the authors synthesize it with a separate-augment pipeline:

A pretrained source separation model (SCNet) splits songs into four pseudo-stems.
Cross-song mixing draws each stem from a different song.
Each stem is independently augmented with random effects from MultiAFx.

MultiAFx is a toolkit they built that unifies 85 effects across 7 Python libraries. Training uses about 105,000 songs from FMA, 1 to 10 random effects per stem, 60 epochs, about 56 GPU-hours on an RTX 5090.

Results

Mixing style retrieval (from 500 candidates, retrieve the one sharing the query's FX chain but with different content):

Chain length	StemFX Top-1	AFx-Rep	Fx-Encoder++	BSFiLM-CL
1 effect	16.8%	3.0%	n/a	n/a
8 effects	86.8%	68.4%	38.0%	77.8%

At 8 effects StemFX reaches 86.8% Top-1 and 0.903 MRR, leading every baseline. The most telling comparison is BSFiLM-CL: same architecture and data as StemFX, only the objective changes from FX chain prediction to contrastive learning, and Top-1 drops 9 points to 77.8%. The prediction objective, not the architecture, is doing the work.

Paired style transfer (moving a target mix's style onto a clip):

Method	MRSTFT (synth)	MUSHRA	Time per example
StemFX	2.35	60.6	0.24s
ITO + AFx-Rep	3.71	30.6	1033s
Target mix (reference)	0.00	96.6	n/a

StemFX has the best spectral fidelity (lower MRSTFT is better), the highest listener preference (MUSHRA 60.6) among methods, and takes 0.24 seconds per example, more than 4000 times faster than iterative optimization (ITO + AFx-Rep at 1033 seconds).

Ablations (8 effects): full model 86.8%; removing FiLM drops to 74.4% (down 12.4 points); also removing source separation drops to 48.0%; swapping in an HTSAT encoder gives 60.2%. Data scale matters too: 10% of data gives 58.6%, 50% gives 77.0%, 100% gives 86.8%.

Why it matters

This turns mixing style transfer from an expensive iterative optimization into a single forward pass, 4000 times faster, and more accurate and more preferred. The separate-augment pipeline that sidesteps multitrack data scarcity is reusable for any audio task short on labels. The same-architecture retrieval comparison also yields a more general lesson: to get a good audio style representation, having the model predict a target beats having it do contrastive learning. For anyone building auto-mixing, DAW plug-ins, or mixing assistants, this is a practical route.

Limitations

The authors concede several points: it can only predict effects seen in training, so adding new effects means retraining; it handles only the four stems that current separation systems produce, and finer professional sessions are future work; pseudo-stems inherit the separation model's errors, and the authors do not quantify how those errors propagate; training chains are sampled randomly, whereas real engineers build structured, genre-dependent chains; and cross-song mixing can produce disharmonious combinations of different keys and tempos. That last point bears on evaluation: if the training data is itself sometimes inharmonious, it is unclear how much of the high listener preference comes from the model learning real mixing logic versus the test set carrying the same kind of noise. The paper does not separate the two.

Terms

FX chain: An ordered sequence of effects and their parameters applied to one stem; the core carrier of mixing style.
stem: A separated track such as vocals, bass, drums, or other.
FiLM: Feature-wise linear modulation, scaling and shifting a network's channels with a conditioning vector.
MRSTFT: Multi-resolution short-time Fourier spectral distance between generated and target audio; lower is better.
MUSHRA: A subjective listening-test scoring method; higher means closer to the reference.