Steering dense music retrieval with open-vocabulary concept discovery
Julien Guinot, Alain Riou, Elio Quinton, György Fazekas
cs.SD, eess.AS
2026-08-09
Training-free sparse inversion maps free-form text to audio SAE neurons. Versus cosine probing, probe AUROC rises from ~50% to 65–80%, with a stronger edit-preservation trade-off.
Dense music retrieval embeds tracks and queries into a shared space such as CLAP or MuQ, then returns nearest neighbors. What users often want is a second edit: more piano on this ambient electronic track, or similar songs without guitar. Changing the whole query moves every attribute at once.
Sparse autoencoders (SAEs) split a 512-d audio embedding into thousands of sparse neurons that look like sliders. The bottleneck is concept attribution. Given a free-form phrase such as jazzy or female vocals, which neurons should move? The default recipe is Discover-Then-Name (DTN): decode each neuron, score cosine against the concept text embedding, pick the closest atom. Two cracks show up. Concepts often split across several neurons (feature splitting). The text-aligned atom may not be the one that fires on audio examples of that concept, because of the modality gap. Edit the wrong neurons and the retrieval either barely moves or drifts.
Universal Music Group and Queen Mary University of London recast attribution as sparse inversion. SAEs are trained on audio only. Ten-second JamendoMaxCaps clips go through the CLAP and MuQ audio towers to 512-d vectors. A BatchTopK SAE expands to 4096 latents (8×), with L0 in {5, 10, 20, 50, 100}. Training is 50k AdamW steps at 1e-4 on one L40, encoder and decoder weights tied. TopK and Matching Pursuit SAEs behaved similarly; the paper reports BatchTopK.
Given a concept text embedding zc, the method does not rank neurons one by one. It solves for a sparse code u whose decode sits close to zc in cosine distance, with a Mahalanobis term estimated from 10k held-out audio embeddings that pins the solution to the audio manifold (γ = 10^{-4}). Two solvers: Adam, initialized from the sparse code of the nearest audio neighbor of zc (otherwise it fails to converge), and FISTA, which exploits the linear decoder. After optimization, IDF weights down generic neurons that fire on almost every track. The same IDF is applied to DTN for a fair comparison. No paired audio-text supervision, no SAE retraining.
Inverting through the full audio encoder costs about 20 seconds per concept on CPU. The Mahalanobis shortcut lands at about 20 ms, which is usable for interactive retrieval. On an internal taxonomy of roughly 300 short tags (top-50 in genre, mood, instrument, usage, era), DTN cosine histograms are mostly Gaussian with a shoulder of positive outliers. Those outliers are almost never a single neuron. After a 4σ cutoff, support sizes still vary widely. DTN is already picking a bundle. The bundle is just the wrong one for audio.
Evaluation uses the MaxCaps validation split, the 20 most frequent tags, and concepts with at least 10 positives. Inverse supports overlap more with the neurons that actually fire on tagged audio.
| Method | Concept-probe AUROC | Bundle recall |
| DTN cosine probing | 50%, near chance | baseline |
| Adam / FISTA inversion | 65–80% | 10–70% higher across sparsities |
Bundle hit rate is higher too: more positive clips activate at least one neuron in the recovered bundle. The probe is a one-hidden-layer MLP of width 512, trained only on the restricted activations. It tests whether those neurons carry concept evidence; it is not a classification leaderboard. In Figure 3, Adam and FISTA sit on top of each other, both well above DTN.
Steering then edits the sparse code. Amplification adds α times the concept direction; suppression subtracts and clips at zero. The result is decoded, L2-normalized, and used for nearest-neighbor search in 10k test clips. Bundle size K is 5, 10, or 20. Edit quality is Δρc@100 (change in the fraction of top-100 neighbors tagged with the concept) and the change in an audio tag classifier's probability. Preservation is a co-occurrence-weighted Jaccard over non-target neighbor tags, plus an audio-versus-text classifier that asks whether the edited vector still looks like audio. Tags that naturally co-occur with the target (electric guitar bringing more rock) are downweighted so plausible side effects are not penalized.
On Pareto frontiers over the top-20 concepts, inversion sits above DTN: at matched off-target drift or manifold adherence, neighbor and classifier gains are larger. In the piano-amplification example, the seed is a C-major ambient electronic clip at about 89 BPM. At α=0.3, DTN drifts to cinematic strings in Fm at 120 BPM. FISTA inversion retrieves a piano-led neighbor still labeled ambient, tempo down to 77 BPM. DTN often walks toward the text manifold. Inversion stays closer to audio. The paper does not table a single Δρc@100 number; the comparison is a binned 90th-percentile frontier.
This adds sliders on top of an existing music-text embedding. It does not retrain the retriever. Any free-form phrase can be inverted into a neuron bundle, with no predefined tag list and no positive audio examples. Twenty milliseconds is fast enough for "less distortion, more vocal" as a live control. For anyone steering SAEs in multimodal spaces, the harder claim is: do not name neurons by text cosine; reconstruct the concept in the target modality.
The gain is still incremental. The control surface is quasi-linear. Quality is capped by CLAP/MuQ and by SAE reconstruction. There is no listening test, and no comparison against re-encoding the query in text or against example-based steering vectors.
The authors say it themselves: weakly grounded, rare, abstract, or cross-modally diffuse concepts inherit SAE reconstruction error, feature splitting, and feature absorption. Sparsity and γ remain hyperparameters, even though a sweep led them to freeze γ at 10^{-4}. Some (concept, K, α) settings still fail. Optimal bundle size is concept-dependent and left open.
The evaluation concepts are dataset tags, not truly open text. The 50% to 80% AUROC is a weak probe, not the retrieval metric. The Pareto plot bins at the 90th percentile and then smooths, so single-point numbers are not recoverable. There is no listening or click study, so product usefulness is untested. Universal Music Group is an author affiliation; the tag taxonomy has an in-house catalog flavor, and transfer should be discounted.