LAION-Audio-XL: A Web-Scale Corpus and Scaling Studies for Contrastive Language-Audio Pretraining
Gijs Wijngaard, Felix Förster, Ross Wightman, Marianna Nezhurina, Christoph Schuhmann, Jenia Jitsev, A. Sophia Koepke
cs.SD, cs.CL
2026-11-09
LAION distilled 1.48B scraped audio clips into 365M captioned pairs and trained CLAP2 from scratch; the 340M CLAP2-L reaches 53.6% on MAEB-30, ahead of 7.7B omni embedding models.
Contrastive language-audio pretraining (CLAP), the audio counterpart of CLIP, has improved mainly by scaling training data. Open audio-text corpora never followed: general-sound caption datasets top out around 50M pairs, and the largest open release, Bagpiper's pretraining collection, is 392M pairs that are mostly speech, all captioned by a single model. Newer audio foundation models report hundreds of millions of hours of private data nobody outside can inspect. Vision and language have LAION-5B and DataComp; audio had no equivalent.
The second open question is allocation. With a fixed compute budget, do you train a bigger model or show a smaller one more samples? Language models and native multimodal models each have an answer; contrastive audio-language training had none.
Half corpus engineering, half scaling study.
The corpus aggregates 1.48B audio samples from 425 sources, 316 of them from or derived from academic releases. YouTube dominates: 32.66M Creative Commons videos cut into 1.18B samples, plus 423M unique audio links scraped from Common Crawl. The pipeline:
The release keeps the 365M pairs of at most 30 seconds, 1.39M hours in total.
CLAP2 spans five sizes from 27M to 340M parameters, all trained from scratch. The audio tower is a NaFlex (NaViT-style) vision transformer over log-mel patches that accepts variable-length audio; the text tower is a ModernBERT-style encoder with RMSNorm and SwiGLU. Training from scratch removes confounds from pretrained weights, which is what makes the scaling curves readable, and batches are bucketed by audio length to cut padding.
| Model | Params | MAEB-30 mean |
| CLAP2-L | 340M | 53.6% |
| CLAP2-M | 153M | 52.3% |
| LCO-Embedding-Omni-7B | 7.7B | 53.5% |
| jina-v5-omni-nano | 855M | 51.8% |
| GLAP | 855M | 44.1% |
| LAION Larger CLAP | 194M | 36.2% |
CLAP2-L tops the table at 340M parameters, one decimal ahead of a 7.7B omni embedding model; below 1B, CLAP2-M at 153M passes the previous best, jina-v5-omni-nano at 855M. Against fellow contrastive audio-text models the gap is wide: 9.5 points over GLAP and 18.4 over LAION-CLAP's 35.2%. The paper also reports that XXS through M lose only to models with at least 8x their parameters. On HEAR, CLAP2-S through L score 81.4-81.9, above every other audio-text model (at most 81.2), though audio-only encoders such as USAD 2.0 still reach 86.8. On the speech-heavy MSEB, CLAP2-L's 30.3 is second behind M2D-CLAP at 30.7.
The scaling numbers carry more operational value:
Audio research gets its first fully documented billion-sample pool and a 365M-pair public corpus, with redistributable audio released directly, links, captions and metadata for the crawled portion, plus checkpoints and code. Teams doing audio retrieval, data curation or text-to-audio generation can use CLAP2 as a drop-in embedder: the 27M tier is cheap enough for filtering, the 340M tier competes on leaderboards.
The allocation finding is directly actionable: on a fixed budget, buy samples before parameters, and cap clips at 30 seconds. Nothing here is architecturally new; NaFlex and ModernBERT are off-the-shelf parts, and the gains come from data engineering.
The authors themselves call the exponents indicative rather than definitive: five discrete sizes, only three interior optima on the Pareto frontier, both ends pinned at boundaries. Extrapolating to 10^12 GFLOPs calls for 1.8B samples, about five epochs over the 365M corpus, while repeated data in language models loses value beyond roughly four epochs; that assumption is untested for audio.
Other weak spots: 94% of captions come from a single generation pass of two models, with no human audit at scale; input is capped near 30 s and evaluation audio is truncated to match, leaving long-audio tasks uncovered; 86.2% of training samples come from YouTube alone; the sound domain saturates; and MSEB leadership belongs to M2D-CLAP, not CLAP2.