AudioCodex: A Fully Causal Continuous Audio Autoencoder at ~10 Hz, Decoding 2x Faster Than Real Time

AudioCodex: Streaming Autoencoders for Live Audio

Marco Pasini, Javier Nistal, Cyran Aouameur, Stefan Lattner, George Fazekas

cs.SD, cs.LG, eess.AS

2026-11-09

AudioCodex is a causal, continuous stereo audio autoencoder at ~10 Hz; its gFAD of 3.70 beats three non-causal baselines, and streaming decode runs about 2x real time on an A100.

What problem this solves

Real-time generative audio (accompaniment, conversation, interactive synthesis) needs a compressor that turns waveforms into short sequences a generative model can work on. Existing general-purpose audio autoencoders fall short in two ways. Most are bidirectional and non-causal, so they cannot encode and decode as audio streams in. The causal ones are mostly speech-oriented and emit discrete tokens, which fit poorly with flow and diffusion models that want continuous latents.

The usual tension sits on top of that: latents rich in detail are hard for downstream models to learn, while heavily compressed ones sound worse. AudioCodex tries to improve that trade-off while staying strictly causal.

Method

Two stages, trained independently.

Training data is Unsupervised People's Speech, Freesound and FMA at 40/30/30. The magnitude model trains for 500k steps at batch size 72 on 3 A100s.

Results

Downstream quality is measured by gFAD (lower is better) using a small latent rectified-flow transformer; reconstruction uses rFAD.

ModelrFADgFAD
AudioCodex dz=1280.7733.696
AudioCodex dz=2560.5934.168
AudioCodex dz=5120.5446.259
CDC (continuous mode)0.3045.065
M2L0.9435.241
SAO1.1519.518

On gFAD, AudioCodex at dz=128 and 256 beats all three non-causal baselines. On rFAD, CDC's 0.304 is lower than every AudioCodex setting, so the claim of matching or exceeding almost all baselines needs a caveat. On spectrogram L1, dz=256 scores STFT 0.098 and MEL 0.065 against CDC's 0.114 and 0.079.

Ablation highlights (rFAD/gFAD): a complex-STFT consistency model gets 2.17/4.81, while magnitude-only Mean Flow gets 0.82/3.61. Adding REPA gives 0.79/3.45. Uniform noise augmentation pushes rFAD to 0.99. Sweeping dz from 64 to 512 moves rFAD from 1.29 down to 0.61 and gFAD from 2.58 up to 4.64, a clear frontier. For the phase estimator, both discriminators together reach 0.25 rFAD, against 0.41 or 0.31 with either alone.

Speed on an A100 (48 kHz stereo, torch.compile): one streaming latent frame encodes in 6.80 ms (12.56x real time) and decodes, phase estimator included, in 41.38 ms (2.06x). For a full 10 s clip, encoding runs at 528.8x and decoding at 114.1x real time.

Why it matters

For anyone building real-time music or voice generation, this is a rare causal autoencoder with continuous latents that plugs straight into flow-style generators. Training three latent sizes jointly lets users pick reconstruction or generation as the priority.

Streaming decode has only about 2x headroom, though. Once a generative model runs alongside it in a live pipeline, the budget gets tight.

Limitations

Terms

Source

What people are saying

Related papers

All paper explainers