SoundStream: An End-to-End Neural Audio Codec
Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, Marco Tagliasacchi
cs.SD, cs.LG, eess.AS
2021-07-07
SoundStream trains a causal conv codec with residual VQ. At 24 kHz, 3 kbps beats Opus at 12 kbps; one dropout-trained model covers 3–18 kbps and runs real-time on a Pixel 4.
Audio codecs used to split into two camps. Waveform codecs such as Opus and EVS reconstruct samples and sound fine at mid-to-high bitrates, then fall apart once the budget drops to a few kbps. Parametric codecs go the other way: they fit a speech production model, stay intelligible at low rates, and fail on music or general audio.
Neural codecs around 2021 could already do speech near 3 kbps. Lyra is the canonical example: hand-crafted mel features in, autoregressive WaveGRU out. Nobody had one model that covered speech, noisy speech and music, scaled across bitrates, and still ran in real time on a phone. That is the gap SoundStream goes after.
The network is three causal-convolution stages. Architectural latency is exactly the encoder downsampling ratio.
Training mixes an adversarial hinge loss, an L1 feature-matching loss inside the discriminators, and a multi-scale mel reconstruction loss. One discriminator family looks at the waveform at several resolutions (the MelGAN pattern); another looks at the real and imaginary STFT. Codebooks follow the VQ-VAE-2 exponential-moving-average recipe, with k-means init on the first batch and replacement of unused entries.
Bitrate scalability comes from quantizer dropout: each training example keeps only the first nq stages, chosen at random. At inference you pick nq for the target rate. Because stages add rather than concatenate, the embedding width never changes, so neither encoder nor decoder needs a redesign.
Denoising is optional. A FiLM layer, placed just before or just after quantization, is driven by a one-hot switch. Clean speech and music reconstruct as-is even when the switch is on, so environmental sound is not stripped by accident. No extra buffering, so no extra latency.
Subjective tests used a crowdsourced MUSHRA-style protocol: 200 clips of 2–4 seconds, 20 ratings each, spanning clean, noisy and reverberant speech plus music. Exact MUSHRA means are in figures, not tables. The paper does report how much bitrate the baselines need to match SoundStream.
| Comparison | SoundStream | Bitrate the baseline needs to match |
| vs Opus | 3 kbps | ≥12 kbps, about 4× |
| vs EVS | 3 kbps | ≥9.6 kbps, about 3.2× |
| vs Opus / EVS | 6 kbps | about 2.6× / 2.2× |
| vs Lyra | 3 kbps | SoundStream wins at the same rate |
Music at 3 kbps still beats Opus at 12 kbps and EVS at 5.9 kbps. Clean and noisy speech stay in the same quality band.
ViSQOL falls smoothly from 3 kbps to 18 kbps and stays above 3.7 at the bottom. Replacing the learned encoder with a fixed mel filterbank drops ViSQOL at 6 kbps from 3.96 to 3.33, worse than cutting the rate in half (learned encoder at 3 kbps is 3.76). A single dropout-trained model matches bitrate-specific models at 6 and 12 kbps, and is only slightly behind at 3 kbps.
On a Pixel 4 single CPU thread, the 8.4M-parameter default (32 channels both sides) encodes and decodes at RTF > 2.3×. Shrinking the encoder to 8 channels raises encoder RTF to 18.6× with almost no ViSQOL loss; shrinking the decoder hurts more. Joint denoising on VCTK at 0 dB SNR scores 2.93 ViSQOL, close to a SEANet-then-SoundStream cascade at 3.05, at about half the compute and with no stacked latency.
Most neural audio tokenizers since then follow this recipe: convolutional encoder/decoder, RVQ, adversarial plus spectral losses. EnCodec, DAC, Mimi and Voxtral Codec all sit on this skeleton. Three design bets got nailed down: the encoder has to be learned, the codebook has to be residual, and one model can cover a bitrate ladder with dropout.
For real-time calls or on-device playback, 13 ms of architectural latency plus phone-CPU headroom is more usable than sample-by-sample autoregressive decoders. Joint compression and denoise also removes one stage from a far-field speech pipeline.
This is a 2021 foundation paper. It is not a 2026 1 kbps SOTA card. Read it to see where later codecs inherited their defaults.
Failure cases are almost absent. The claimed correlation between ViSQOL and listening tests is described as “strong in early experiments,” with no coefficient. The MUSHRA variant drops the low-pass anchor, so absolute scores do not travel cleanly across labs.
Rate is constant; there is no cross-frame entropy coder. Symbol entropy suggests another 7%–20% could be saved, and that coder was never shipped. Training data is LibriTTS, Freesound-mixed noise and MagnaTagATune, not large-scale in-the-wild audio. Joint enhancement is only background noise suppression.
The Opus/EVS comparison is 24 kHz listening quality, with no packet loss or jitter-buffer numbers. Real-time figures are Pixel 4 single-thread only.