SonicCaps: Large-Scale Diverse and Fine-Grained Captioning for Improved Audio-Retrieval
Zineb Lahrichi, Marc Ferras, Gaël Richard, Geoffroy Peeters
cs.SD, cs.CL, cs.MM, eess.AS
2026-09-02
Qwen3-Omni writes ~24 captions for each of ~700k clips (~15M total). Mixing those styles to train CLAP lifts AudioCaps text-to-audio R@5 from 64.7 (LAION) to 79.5.
Audio-language pretraining has been bottlenecked by captions. Human-written sets such as AudioCaps and Clotho stay in the tens of thousands. WavCaps and LAION-630k scale up by rewriting tags and noisy metadata with language models, then the sentences turn generic and repetitive, and most corpora still pair each clip with one or two texts.
Hearing is many-to-many. The same page-turn can be "crisp paper in a quiet room" or "soft shuffling." Current datasets barely use that ambiguity. Pipelines like WavCaps never listen to the audio; they filter away nearly half of FreeSound to suppress noise, and what remains still collapses to templates such as "Music is being played."
Sony CTC and Telecom Paris keep the audio corpus fixed and ask a narrower question: if each clip gets many captions that actually describe the sound, does CLAP retrieval move. CLAP is CLIP for audio, a contrastive map from waveforms and sentences into one vector space.
SonicCaps is about 700k clips and about 15M captions, drawn from FreeSound (515k), the BBC Sound Effects archive (31k), AudioSet Strongly-Labeled (108k), and AudioCaps (49k). The public release is captions plus audio IDs. The waveforms stay with their original hosts.
Captions come from Qwen3-Omni-30B-A3B-Instruct, conditioned on both the clip and the original text. Audio is resampled to 16 kHz and truncated to 10 seconds. Decoding uses temperature 0.6, top-p 0.95, and a 30-token cap. Three stages:
The four types average 23.9 captions per clip. Vocabulary grows from 26k in main to 66k overall. On FreeSound, unique captions rise from 46.7% (raw) and 80.4% (WavCaps) to 92.4%. The most frequent caption appeared 36,773 times in the raw dump and 317 times in SonicCaps. WavCaps had cut FreeSound from 515k to 250k; this pipeline keeps the extra half.
Downstream CLAP is frozen as an architecture: PaSST audio encoder, RoBERTa-Large text encoder, 1024-d joint space, symmetric contrastive loss, temperature 0.2, effective batch 448. Training samples from the original captions and the four SonicCaps types by a mixing schedule, and drops punctuation with probability 0.2. The public SonicCLAP checkpoint further drops incompatible FreeSound licenses, leaving 371k clips, still more audio than WavCaps' 250k.
A listening study with 25 people and 375 ratings compared pairs on 60 clips. SonicCaps MOS sits above human AudioCaps captions, raw FreeSound text, and WavCaps. About 50% of SonicCaps captions drew no negative checkbox at all, versus under 25% for the other sources. Captions that were globally unrelated to the audio showed up only in WavCaps and raw FreeSound.
When the audio stays fixed, retrieval gains track caption diversity. Training on the human-preferred main captions alone barely beats the original mixed captions.
| Model | AudioCaps T2A R@5 | Commercial T2A R@5 |
| LAION-CLAP | 64.7 | 23.5 |
| Self-trained AC+WavCaps | 66.8 | 18.1 |
| SonicCaps main only | 71.1 | 23.0 |
| Equal mix SonicCLAPAR | 79.5 | 32.4 |
AudioCaps T2A R@10 moves from 75.8 (LAION) to 91.3. Mixing paraphrases, short queries, and tags is what lifts the curve. A plot of caption sampling perplexity (the effective number of captions a clip can draw in training) shows R@5 rising with that number.
Zero-shot classification follows the same split. ESC-50 overlaps FreeSound, so treat those numbers as leaky: LAION R@1 is 82.1, the diverse mix Ours(8) reaches 89.4. On fully disjoint FoleyBench the gap is clearer. LAION R@5 is 9.14; SonicCLAPAR reaches 25.6.
A second checkpoint, SonicCLAPMOS, is trained only on main captions. Spearman correlation between MOS deltas and CLAP-score deltas is -0.07 for LAION and 0.32 for this model. As a proxy for caption quality, LAION-CLAP scores are close to useless on this listening test.
For audio retrieval, SFX search, and zero-shot tagging, the reusable move is the supervision recipe: keep the audio, write each sound at several granularities and sentence styles, and let contrastive learning see the one-to-many map. Short lines look like real queries, tags look like class names, long lines look like catalog copy. Mixing them beats a single "high-quality" sentence.
The dataset is on Hugging Face, with two checkpoints: SonicCLAPAR for retrieval and SonicCLAPMOS for perceptual scoring. Audio is not in the dump; users pull it by ID from FreeSound, AudioSet, and BBC.
No new encoder. The claim is about data and sampling. Caption fidelity matters to listeners. For retrieval it is a small effect. Diversity is the driver.
The authors flag several gaps. Qwen3-Omni still misses tiny timbre differences, so collections of hi-hats collapse to near-duplicate sentences. Prompts are soft constraints; hallucinations and template phrasing shrink, they do not vanish. Inference audio is cut at 10 seconds, so the tail of a long recording never enters the caption. Because waveforms are not released, reproduction cost sits in crawling and licenses, not in a single download.
The experimental seams are real. The commercial val set is 500 pairs from each of three internal SFX libraries; outsiders cannot rerun it. ESC-50 overlaps FreeSound training data, which the paper admits inflates scores, hence FoleyBench. Even there the best R@1 is 9.66, far from a usable classifier. The listening test covers 60 clips and 25 raters, and the MOS figure never states exact means in the text. The CLAP skeleton is locked, with no same-data swap against HTSAT, so the gap versus LAION-CLAP mixes encoder choice with caption choice. AudioSetCaps (6M pairs), Auto-ACD, and Sound-VECaps never enter the same training bake-off.