Kandinsky 6.0 Video: Foundation Models for Synchronized Video and Audio Generation
Team Kandinsky, Julia Agafonova, Bulat Akhmatov, Mikhail Aksyutin, Grigorii Alekseenko, Anastasia Aliaskina, Olga Androsova, Vladimir Arkhipkin, Anna Averchenkova, Alexander Belykh, Serafima Bocharova, Sofiya Bogakovskaya, Anton Bukashkin, Mark Bulygin, Kirill Buzygin, Irina Cheremnykh, Kirill Chernyshev, Mikhail Chernyshov, Vladimir Chernyy, David Chikovani, Georgy Daniltsev, Denis Dimitrov, Anna Dmitrienko, Vladimir Dokholyan, Sergey Emelyanov, Dmitry Ermilov, Georgii Fedorov, Polina Gavrilova, Nikolai Gerasimenko, Aleksandr Gordeev, Andrey Inozemtsev, Andrei Ivaniuta, Alexander Ivanov, Mikhail Karaev, Anastasiia Kargapoltseva, Ivan Kirillov, Nikita Kiselev, Valeria Kobenko, Yury Kolabushin, Denis Koposov, Anatoly Korobov, Vladimir Korviakov, Kirill Kozlov, Denis Krzhivokolskiy, Konstantin Kuklev, Alexander Kunitsyn, Sergey Kuzin, Vladislav Lakhtionov, Alexey Letunovskiy, Maxim Litvinov, Alexander Lyulkov, Georgy Makarov, Kirill Malakhov, Egor Malykh, Mikhail Mamaev, Dmitrii Mikhailov, Polina Mikhailova, Ivan Mikheev, Elizaveta Muromtseva, Nikolai Nazarkin, Tatiana Nikulina, Lev Novitskiy, Stanislav Onuchin, Nikita Osterov, Denis Parkhomenko, Anatoliy Parpara, Vladimir Polovnikov, Konstantin Reznikov, Azat Saginbaev, Nikita Samsonov, Alexander Sentsov, Nikita Shaimov, Artem Sherstyuk, Andrey Shutkin, Egor Silvestrov, Bulat Suleimanov, Matvey Suprunov, Sergey Taranov, Irina Tolstykh, Tatiana Trofimuk, Ilya Trushkin, Aleksandra Tsybina, Olga Varlashina, Viacheslav Vasilev, Ilya Vasiliev, Eugeny Vilisov, Sergey Yakubson, Konstantin Zakharov
cs.CV, cs.AI, cs.LG, cs.MM
2026-10-05
Kandinsky 6.0 (Lite 3B, Pro 29B) generates 5-second, 44 kHz clips with lip-sync using a dual-stream CrossDiT. RL cuts Pro speech WER from 0.235 to 0.124.
Veo 3.1, Sora 2, and Wan 2.6 already turn out short clips with speech and ambient sound, and they do not ship weights. Open models still trail on duration, resolution, and how tightly sound sits on the picture. Kandinsky 5.0 got video down to a trainable CrossDiT with NABLA sparse attention. It had no audio stream trained alongside the video.
Joint generation has to land semantics, timing, and lip shape at once. Hang an audio head on a finished video model and the picture usually degrades, while the audio branch has never had a pretraining run of its own. Kandinsky 6.0 keeps the 5.0 video stream, trains audio from scratch, and ties the two with bidirectional cross-attention. The release target is a 5-second clip at 44 kHz with lip-sync, under an MIT license.
The network is an asymmetric dual-stream CrossDiT. Each stream runs its own self-attention and reads text on its own. Between streams, every block has two cross-attention directions: video queries read audio keys and values (V2A), and audio queries read video (A2V). Pro is 29B parameters, split into a 19B video stream, a 5B audio stream, and 5B of cross-attention, with 60 blocks in each stream. Lite is 3B, split 2B, 0.6B, and 0.4B, with 32 blocks per stream. Video latents come from the Hunyuan VAE, audio from the MMAudio autoencoder at 44 kHz, text from Qwen2.5-VL-7B-Instruct and CLIP. Video RoPE indexes frames after normalizing frame rate against 24 fps.
The training objective is flow matching: the network fits the velocity from noise toward data, and the loss is MSE on that velocity.
Paired clips with people are kept only if the lip-sync score LSA is at least 3 on a 0–5 scale, and event-level Desync stays under 0.1. About 25% of the set has no visible face and skips that cut. The image-conditioned set has 116,000 samples. The reference frame is concatenated on the time axis and marked with a binary mask. A learned token-role embedding barely moved inference outputs. RoPE is what binds the reference frame to a time index.
Supervised fine-tuning trains one model on each of 11 domains, then averages the weights into a model soup. Stage 1 uses 4,663 clips, freezes audio, and optimizes video only at 1e-5 for 7,000 steps. Stage 2 uses 7,686 clips with more speech and music. Video stays at 1e-5, audio drops to 1e-6, and the loss is 0.85 on video, 0.15 on audio, plus 1.0 on the face region of the video latent, for 5,000 steps before the average. Both stages are 75% text-to-audio-video and 25% image-conditioned.
RL runs on text-to-audio-video only and is adapted from OmniNFT. SFT weights stay frozen. LoRA, a low-rank update on linear layers (r=16, α=32), touches 199M parameters on Pro (0.69%) and 47M on Lite (1.57%). Each prompt yields 10 clips on Pro and 8 on Lite, at 512×768, with a deterministic DPM2 sampler, 20 steps, and CFG 1.0. Diversity inside a group comes from the seed. Eight rewards are grouped as video (weight 2.5: HPSv3, VideoAlign), audio (2.0: AudioBox, CLAP, Whisper-ASR), and sync (2.75: AV-desync, AV-align, and a LatentSync score gated by speech intelligibility). Sync terms enter both modality advantages. Speech clips are oversampled 1.5×. Gradients on the V2A key-value path are scaled down by layer so video updates do not wipe the audio representation. A KL term toward SFT uses coefficient 1e-4. The checkpoint is the best five-epoch sliding mean of the reward, not the last epoch.
Distillation targets 10 network evaluations (NFE). Distribution-matching methods (DMD, DMD2, SID) mode-collapsed in these runs and spent more on teacher forwards. π-Flow imitates the guided teacher velocity on states the student actually visits, so classifier-free guidance does not need its own distillation stage. Fewer steps wipe high-frequency texture. Sim-LADD puts an adversarial loss on the student's backward rollout and backpropagates through the last 5 of the 10 steps, for 300 updates.
Full HD is a second model. K-VAE compresses space by 16 and time by 4. A latent upscaler raises resolution in that latent, then a text-free SR-DiT of about 1.41B parameters removes degradation and is distilled to 2 evaluations. The 1080p path bilinearly upscales by 1.125 in pixels and then takes the ×2 route. The optional high-quality first-frame anchor is zero at inference.
Word error rate is the number with a direct before-and-after. On a held-out speech set of 64 prompts and 3 seeds, scored by Whisper-large-v3 with a paired Wilcoxon test:
| Setup | Baseline | WER |
| Pro after RL | SFT 0.235 | 0.124, 47% relative drop, p<0.001 |
| Lite after RL | SFT 0.179 | 0.127, 29% relative drop, p<0.001 |
| Pro, 100 Harvard sentences | SFT 0.251 | 0.180, p=0.005 |
Whisper-large-v3 is also a reward model, so the 47% figure shares a judge with the training signal. The Harvard drop is smaller and still significant.
Side-by-side human evaluation, as the introduction summarizes it: Pro clearly beats Kandinsky 5.0 Video Pro, beats open LTX 2.5 on most VABench metrics, and stays competitive with Kling 2.6, Veo 3.1 Fast, MiniMax H3, and Seedance 2.0, especially on speech. The introduction does not publish per-criterion win rates.
Super-resolution has no paired ground truth. Sources are 5-second clips generated by Kandinsky 5.0 Video Lite, 912 crops per scale, so absolute scores should not be compared across scales.
| Scale | Stage | MUSIQ | DOVER | Laplacian artifacts | Warp error |
| ×4 | dense attention | 48.92 | 0.5467 | 0.4366 | 0.6141 |
| ×4 | after distillation | 50.41 | 0.5553 | 0.4227 | 0.6781 |
| ×2 | dense attention | 66.05 | 0.7329 | 0.6062 | 0.6551 |
| ×2 | after distillation | 65.81 | 0.7348 | 0.5710 | 0.6831 |
At ×4, MUSIQ rises from 48.92 to 50.41 and warp error rises to 0.6781. At ×2, MUSIQ stays between 65.29 and 66.05 across the three stages. CLIP-IQA improves at ×4 only, from 0.4426 to 0.4587.
On 94 held-out clips, the latent upscaler is compared with decode-then-bilinear in pixels. At ×4, mean PSNR is 39.26 dB versus 37.96 dB, ahead in every group. At ×2, 41.70 versus 41.58 overall, but bilinear wins on weakly degraded video (42.97 vs 41.60 on the Kandinsky 5.0 subset). Bilinear interpolation inside the latent scores only 21.44 dB at ×2 and 19.76 dB at ×4. On 24 real clips, warp error rises by 0.026 and 0.049 versus a K-VAE round trip (baseline about 0.591), roughly 4.4% and 8.3%.
Weights, code, and a diffusers integration are MIT-licensed. A lip-synced audio-video base model with that license is still uncommon. Lite is the size for experiments, Pro is the size for quality, and both share 44 kHz audio, lip-sync, and text or image conditioning. Rewards, distillation, and the super-resolution stack can all be changed from the release.
The operating range is five-second speech, ambience, and short visual beats. Long shots sit outside it. Against Kandinsky 5.0 the video backbone is continued training. What this generation adds is synchronization and post-training. The paper places the comparison with Kling and Veo on speech quality, which matches a 5-second SD generator. Treating the result as a lead on duration or resolution goes past what the experiments show.
Clips stay near 5 seconds, and joint training heavily prefers 768×512 landscape. In Table 3 that bucket is sampled at 0.97 or higher. 1080p is entirely the external super-resolution stack. At ×2 the latent upscaler loses PSNR to pixel bilinear on weakly degraded video, and at ×4 warp error grows from 0.6141 to 0.6781. The paper notes that this residual also tracks detail and optical-flow accuracy, and final checkpoints were picked by side-by-side viewing.
The largest WER drop uses Whisper-large-v3 as both reward and metric. Harvard sentences move only from 0.251 to 0.180. RL rollouts use CFG 1.0 at 512×768 and never include image conditioning, so there is no measurement of whether the speech gain survives guided Full-HD inference.
Fine-tuning rests on a few thousand expert-filtered clips, and the 40 million audio tracks were cut from the same in-house video pool. Lip-sync outside that filter is untested. The training section describes texture loss after few-step distillation and the adversarial repair, without a human score against the undistilled model. The first-frame high-quality anchor is unused on the current inference path.