Kandinsky open-sources KVAE, a 3-modality tokenizer family matching Wan, FLUX and MovieGen

KVAE: Family of Tokenizers for Multimodal Generative Models

Andrey Shutkin, Denis Parkhomenko, Ivan Kirillov, Kirill Chernyshev, Kirill Malakhov, Ilia Vasiliev, Ilia Trushkin, Valeriya Kobenko, David Chikovani, Alexander Ivanov, Azat Saginbaev, Egor Silvestrov, Ivan Mikheev, Konstantin Zakharov

cs.CV, cs.LG, cs.SD

2026-08-06

Kandinsky's KVAE: continuous latent tokenizers for audio, image and video diffusion, matching or beating Wan, FLUX, HunyuanVideo and MovieGen at equal compression. Open weights.

What problem this solves

Latent diffusion models for images, video and audio share a first step: a tokenizer compresses the raw signal into a low-dimensional latent space, and the diffusion model generates inside that space. The tokenizer therefore sets a ceiling on generation quality, training speed and what the downstream model can learn. The standard way to judge a tokenizer is reconstruction fidelity, metrics like PSNR, LPIPS and PESQ. This paper rests on the claim that those numbers mislead: high reconstruction does not guarantee good downstream generation, a tension Yao et al. call the reconstruction-generation dilemma. Kandinsky Lab builds three tokenizers (audio, image, video) organized around a second property they call diffusability, meaning how friendly a latent space is to diffusion modeling.

Method

Three continuous-latent tokenizers:

• KVAE-Audio: full-band 48 kHz, compressing audio into a 50 Hz, 64-channel latent (50 tokens per second). It keeps DAC's convolutional backbone with dilated residual units but swaps residual vector quantization for a continuous Gaussian bottleneck and adds one self-attention block at the bottleneck. Snake activation. The loss combines waveform reconstruction, multi-scale spectral, KL, adversarial, perceptual and an alignment regularizer.
• KVAE-2D (image): a four-level convolutional residual autoencoder, 8x8 spatial compression, 32 latent channels.
• KVAE-3D (video): two causal tokenizers, 4x8x8 with 16 channels and 4x16x16 with 64 channels. Conv3D blocks, attention-free, with spatial RMSNorm instead of GroupNorm to preserve causality. Trained on 10M images plus 2M videos in a 4-stage progressive schedule from 65 to 129 frames.

Two design choices matter most. The decoder is deliberately asymmetric: the 4x16x16 model widens the decoder and thins the encoder, giving a 5.3x parameter ratio, because the decoder carries the generative load. And channel count is the sharpest knob trading reconstruction against generation. More channels reconstruct better but make the downstream diffusion model converge more slowly. The report is explicit that 64 channels is the optimum at the 0.6B generator scale, not an intrinsic property of audio latents.

Results

Video reconstruction (MCL-JCV 720p):

At equal compression KVAE is about 1 dB higher in PSNR and leads on SSIM.

Image reconstruction (OmniDoc-TokenBench, 3042 images): KVAE-2D-2.0 reaches PSNR 28.05 and SSIM 0.957, above FLUX.1-dev (26.24) and FLUX.2-dev (27.72). Its FID (1.74) is worse than both FLUX models (0.55, 0.73), so reconstruction is not a clean sweep.

Audio is the cleanest result. On reconstruction (AudioSet), KVAE-Audio posts MEL 0.537 and STFT 1.770, beating MMAudio, MovieGen's DACVAE and SAME-L, at 166.9M parameters versus 852.1M for SAME-L. For comparison, MMAudio's VAE reports a near-collapsed SI-SDR of -32.08. On generation (AudioCaps) KVAE-Audio leads on every metric, including CLAP 0.344 plus CE, perceptual quality and FAD, and human side-by-side win rates against every baseline sit between 0.54 and 0.74.

Why it matters

Kandinsky Lab ships weights, training recipes and ablations for all three tokenizers, which is a ready blueprint for anyone building their own latent diffusion. Numerically the family sits alongside or above frontier open tokenizers like Wan-2.2, HunyuanVideo, FLUX.2, MovieGen, StableAudio and MMAudio. More useful than the leaderboard is the methodological message: pick tokenizers by diffusability, not reconstruction alone, and treat channel count and encoder-decoder symmetry as explicit reconstruction-generation trade-offs. Those are lessons that only come from training the full stack.

Limitations

Every "matches the frontier" claim comes from the authors' own evaluation on their own downstream generators (0.6B for audio, 2B for image), with no third-party reproduction and no head-to-head against closed systems. On the image side KVAE loses FID and LPIPS to FLUX, so the lead is not universal. The report admits that model-based aesthetic predictors are not independent of the systems they score. And the 64-channel optimum is validated only at 0.6B; how it holds at larger generator scale is untested.

Terms

• tokenizer (here): an autoencoder that compresses image/video/audio into low-dimensional latents; the latent diffusion model then generates inside that space.
• diffusability: how amenable a latent space is to diffusion modeling, a property distinct from reconstruction fidelity.
• reconstruction-generation dilemma: better-reconstructing tokenizers do not always yield better downstream generation; the two tug against each other under fixed compute.
• RVQ (residual vector quantization): encoding a latent as multi-level codebook indices, common in audio; KVAE replaces it with a continuous Gaussian bottleneck.
• causal architecture: processing that uses only past and present frames, never future ones, which suits streaming generation.
• SI-SDR: scale-invariant signal-to-distortion ratio, measuring how close reconstructed audio is to the original; higher is better.

Source

• arXiv 2608.05798v1
• PDF
• Hugging Face
• GitHub · https://github.com/kandinskylab/kvae

What people are saying

• KVAE Tokenizers: Outperforming Frontier Open-Source VAEs Across Image, Video, and Audio NielsRogge

Related papers

• SDS Collapses to Modes; Probability-Flow Distillation Matches the Target Spread
• Staged Blender Code Beats SAM-3D Agents at Single-Image Inverse Graphics
• TAPe+ML claims 88.1% ImageNet and 65.3 COCO AP with under 100k parameters
• Shared hypernetwork synthesizes VideoQA prompts on demand, cutting NExT-QA forgetting to 4.76
• GPT-6 Astra as robot policy hits 22.48% on RoboDojo, beating 40 public VLAs
• RegVGGT keeps 1% of tokens per frame for streaming feed-forward reconstruction

All paper explainers