Ontology-based Target Sound Extraction
Carlos Hernandez-Olivan, Marc Delcroix, Tsubasa Ochiai, Naohiro Tawara, Shoko Araki
eess.AS
2026-09-01
NTT conditions SoundBeam on AudioSet so one model extracts at any ontology level in one pass. Multi-hot plus CPCC hits 7.10 dB leaf SI-SNRi vs 5.42 aggregation, 6.43 vs 1.52 at root.
Target sound extraction (TSE) isolates one source from a mixture given a semantic query. Most class-conditioned systems treat labels as unrelated. "Cat" and "dog" share no structure in the embedding table, and a coarser query such as "animal" is not a first-class input to the same model.
That mismatch shows up in products. A home monitor may ask for all animal sounds; a pet collar only wants the cat. Kong et al. detect active leaf classes, extract each one, then group the outputs with the AudioSet ontology. That pipeline needs a separate detector and one forward pass per leaf, and the TSE trainer never sees the tree. Hyperbolic embeddings have been used to bake hierarchy into source separation, but the geometry is not scored against tree distance.
The reformulation is: any node on a sound ontology can be the query, one model, one forward pass. The tree is a three-level cut of AudioSet. Roots (L0) are animal, human, music, and soundsthings. Intermediate nodes (L1) include domestic, voice, instrument, alarm. Leaves are cat, dog, female speech, and similar fine classes. Some nodes are scored at both intermediate and leaf levels.
The backbone is SoundBeam, a time-domain encoder-masker-decoder. A 1D convolution encodes the mixture; a TasNet-style masker is conditioned by multiplying latents with a query embedding; a decoder rebuilds the waveform. Class information lives in a learned embedding table. Three conditioning schemes are compared.
Leaf one-hot plus aggregation is the baseline. Each leaf is extracted on its own, then all leaves under the queried node are summed. Training uses only leaves that are actually in the mixture. At test time the active set is unknown, so every descendant leaf is extracted and added. The idea is close to Kong et al., minus the detector, so inactive classes still contribute garbage.
Multi-hot conditioning turns on every descendant leaf of the queried node. The parent embedding is the sum of child embeddings, so the hierarchy is written into the query itself. Train and test use the same multi-hot vector. The reconstruction target is the sum of isolated waveforms of descendants that are present. One pass.
Full-ontology one-hot gives every node, leaf or internal, its own embedding. "Animal" is no longer a sum of "cat" and "dog"; it is a separate concept.
Each training batch samples one node per ontology level. Reconstruction is negative SI-SNR. A CPCC term from image classification is added with weight λ=0.1: Pearson correlation between pairwise Euclidean distances of embeddings and shortest-path distances on the tree is maximized. Sibling leaves sit at tree distance 2; a leaf and its parent sit at 1; cross-root pairs go up to twice the tree depth.
Mixtures are synthetic. FSD50K supplies environmental events, LibriSpeech replaces the speech, SynthSOD and MoisesDB fill instruments and classical. Sampling is two-stage: 2-3 leaves from one subtree, then 2-3 from a different root, so every 5-second clip spans at least two roots and holds 3-5 targets of 2.5-3.5 seconds mixed at 5-15 dB SNR against background. Splits are 48K / 4K / 8K. Encoder and decoder use kernel 16 and stride 8; the masker has 512 filters and 8 blocks; embedding size is 256; batch 8; learning rate 1e-5.
Numbers are mean SNRi / SI-SNRi on the test set, in dB. Unprocessed mixtures already sit below the targets: -0.69 at the root, -2.00 at L1, -5.43 at leaves. Root-level input SNR is higher because several events often share a root, so the target is a larger slice of mixture energy.
| Conditioning | CPCC | Root SI-SNRi | L1 SI-SNRi | Leaf SI-SNRi |
| Leaf extract + aggregate (all leaves) | no | 1.52 | 2.97 | 5.42 |
| Leaf extract + aggregate (oracle detection) | no | 3.79 | 4.22 | 5.42 |
| Multi-hot | no | 5.57 | 5.93 | 6.12 |
| Multi-hot | yes | 6.43 | 6.66 | 7.10 |
| Full-ontology one-hot | no | 4.42 | 4.90 | 5.68 |
| Full-ontology one-hot | yes | 4.75 | 5.12 | 5.73 |
All-leaf aggregation still works at leaf level (SI-SNRi 5.42, matching the oracle, because a leaf query extracts one class). At the root, SNRi collapses to -4.67 and SI-SNRi to 1.52, because inactive leaves are extracted and summed. Oracle detection removes that leak and recovers 3.79 SI-SNRi at the root. That row is an upper bound on the detect-then-extract pipeline, not a reproduction of Kong et al.
Multi-hot in one pass already beats that oracle bound. With CPCC it is best at every level: 6.43 / 6.66 / 7.10 SI-SNRi for root / intermediate / leaf. The leaf gain over aggregation is +1.68 dB SI-SNR, and another +0.98 over multi-hot without CPCC. Full-ontology one-hot trails multi-hot by more than 1 dB at root and intermediate levels. CPCC barely helps it; root SNRi even drops from 5.87 to 5.52.
Writing a parent as the sum of its children is the better inductive bias. The tree regularizer pays off in that shared space, not in a table of independent node embeddings.
For computational auditory scene analysis and hearables, the product request is one set of weights that can answer both "pull the speech" and "pull all human sounds" without a detector in front. Multi-hot makes that a single forward pass and cuts inference from O(|L|) in the number of leaves to O(1).
For people training TSE, the leaf numbers say hierarchy helps even if the deployed query is always a leaf: 5.42 to 7.10 SI-SNRi, same SoundBeam backbone, different query encoding and one extra loss. The cost is isolated leaf stems plus ontology-aware mixing. This is an incremental systems paper. The new work is the conditioner and the regularizer.
All audio is 5-second synthetic mixtures with 3-5 sources and SNR held in 5-15 dB. Overlap, reverb, and noisy ontology labels from real recordings are not tested.
The ontology is a closed AudioSet cut. Queries must be nodes on that tree. There is no open-vocabulary text query and no enrollment clip. Both are listed as future work.
λ for CPCC is only 0.1; a sensitivity study is deferred. On full-ontology one-hot, CPCC slightly lowers SNRi at root and leaf, so the term is not a free lunch. It helps when parent embeddings already share child weights.
There is no experimental comparison with hyperbolic source separation. The oracle-detection baseline is a conceptual ceiling, not Kong et al. reproduced. The 8K test clips come from the same synthesis recipe as training, so the generalization gap is unknown.