AudioCodex: A Fully Causal Continuous Audio Autoencoder at ~10 Hz, Decoding 2x Faster Than Real Time
AudioCodex: Streaming Autoencoders for Live Audio
Marco Pasini, Javier Nistal, Cyran Aouameur, Stefan Lattner, George Fazekas
cs.SD, cs.LG, eess.AS
2026-11-09
AudioCodex is a causal, continuous stereo audio autoencoder at ~10 Hz; its gFAD of 3.70 beats three non-causal baselines, and streaming decode runs about 2x real time on an A100.
What problem this solves
Real-time generative audio (accompaniment, conversation, interactive synthesis) needs a compressor that turns waveforms into short sequences a generative model can work on. Existing general-purpose audio autoencoders fall short in two ways. Most are bidirectional and non-causal, so they cannot encode and decode as audio streams in. The causal ones are mostly speech-oriented and emit discrete tokens, which fit poorly with flow and diffusion models that want continuous latents.
The usual tension sits on top of that: latents rich in detail are hard for downstream models to learn, while heavily compressed ones sound worse. AudioCodex tries to improve that trade-off while staying strictly causal.
Method
Two stages, trained independently.
- Magnitude autoencoder (150M parameters): works only on log-magnitude spectrograms. The encoder uses time-causal convolutions plus a per-frame transformer along the frequency axis, with 8 learnable summary embeddings per frame that get pooled into a 10 Hz latent. The decoder is a one-step Mean Flow model (it learns the average velocity between two noise levels, so one evaluation traverses the path), trained with the iMF velocity-regression objective.
- Phase estimator (70M): a causal U-Net trained adversarially with R3GAN. It takes ground-truth magnitude plus noise and predicts phase as cos/sin channels, which avoids wrap-around discontinuities. Phase is split out because it varies quickly and interferes with magnitude learning.
- Sphere bottleneck with bounded noise: latents are RMS-normalized onto a hypersphere. During training noise is interpolated in, with its maximum strength capped by the noise level of the corresponding input spectrogram. A clean input gets only a lightly perturbed latent, which keeps high-frequency detail while making the latent space easier to sample.
- REPA: cosine alignment of one hidden layer each in the encoder, upsampler and generator to a frozen USAD-Small teacher, weight 0.1.
- CFG distillation: for about 10% of each batch, classifier-free guidance is baked into the one-step target.
- Causality: time-causal convolutions and causal group norm that accumulates statistics only from the past; inference caches activations and processes new input incrementally.
Training data is Unsupervised People's Speech, Freesound and FMA at 40/30/30. The magnitude model trains for 500k steps at batch size 72 on 3 A100s.
Results
Downstream quality is measured by gFAD (lower is better) using a small latent rectified-flow transformer; reconstruction uses rFAD.
| Model | rFAD | gFAD |
| AudioCodex dz=128 | 0.773 | 3.696 |
| AudioCodex dz=256 | 0.593 | 4.168 |
| AudioCodex dz=512 | 0.544 | 6.259 |
| CDC (continuous mode) | 0.304 | 5.065 |
| M2L | 0.943 | 5.241 |
| SAO | 1.151 | 9.518 |
On gFAD, AudioCodex at dz=128 and 256 beats all three non-causal baselines. On rFAD, CDC's 0.304 is lower than every AudioCodex setting, so the claim of matching or exceeding almost all baselines needs a caveat. On spectrogram L1, dz=256 scores STFT 0.098 and MEL 0.065 against CDC's 0.114 and 0.079.
Ablation highlights (rFAD/gFAD): a complex-STFT consistency model gets 2.17/4.81, while magnitude-only Mean Flow gets 0.82/3.61. Adding REPA gives 0.79/3.45. Uniform noise augmentation pushes rFAD to 0.99. Sweeping dz from 64 to 512 moves rFAD from 1.29 down to 0.61 and gFAD from 2.58 up to 4.64, a clear frontier. For the phase estimator, both discriminators together reach 0.25 rFAD, against 0.41 or 0.31 with either alone.
Speed on an A100 (48 kHz stereo, torch.compile): one streaming latent frame encodes in 6.80 ms (12.56x real time) and decodes, phase estimator included, in 41.38 ms (2.06x). For a full 10 s clip, encoding runs at 528.8x and decoding at 114.1x real time.
Why it matters
For anyone building real-time music or voice generation, this is a rare causal autoencoder with continuous latents that plugs straight into flow-style generators. Training three latent sizes jointly lets users pick reconstruction or generation as the priority.
Streaming decode has only about 2x headroom, though. Once a generative model runs alongside it in a live pipeline, the budget gets tight.
Limitations
- The non-causal variant is better (rFAD 0.65 vs 0.79, gFAD 3.25 vs 3.45), so the causal constraint has a real cost, as the authors admit.
- The gFAD proxy uses 1.5 s samples and a 4-layer model, so it cannot measure long-range structure. Whether the conclusions carry over to large models and long audio is untested.
- Phase needs a separately trained model, which adds system complexity.
- Multi-step decoding gives no clear metric gain.
- Every baseline is non-causal and no comparable causal continuous model exists, so the state-of-the-art claim rests on a narrow reference set.
- Ablations run only 200k steps and report a four-domain average, hiding per-domain differences.
Terms
- Mean Flow: learns the average velocity between two noise levels, so one function evaluation can cover the whole generation path.
- REPA: Representation Alignment, which pulls a model's hidden features toward those of a frozen self-supervised teacher.
- FAD: Fréchet Audio Distance, a distribution distance between generated and real audio in embedding space; lower is better.
- CFG distillation: baking classifier-free guidance into the model so inference needs one forward pass instead of two.
- RTF: real-time factor, audio duration divided by processing time; above 1 means faster than real time.
Source
What people are saying
Related papers
All paper explainers