DreamX-Creator: Democratizing Native Audio-Video Generation at 2K Resolution
Jiashu Zhu, Yanhao Zheng, Ruitian Tian, Rujing Dang, Shen Zhang, Bingze Song, Jiachen Lei, Ruimin Lin, Jiahong Wu, Xiangxiang Chu
cs.CV, cs.SD
2026-09-01
7B joint audio-video generator with gated cross-modal attention and a 1-step 2K refiner. Beats most open baselines on DeSync; trails larger systems on audio aesthetics.
Most video generators either skip audio or add it after the fact. Speech, impacts, ambience and camera motion are jointly determined; a cascade cannot model that reciprocity. Native joint systems exist, but almost none combine downloadable weights, a local 2K path, and a backbone under 10B. LTX-2.3 is 22B, MOVA is 32B/18B, MAGI-2 Preview is 114B/6B. DreamX-Creator 1.0 ships a 7B joint generator plus a local 2K refiner.
The generator takes a first frame and a text prompt, then jointly denoises audio and video latents with flow matching. The first half of the network keeps the two streams independent. The latter half opens paired audio-to-video and video-to-audio cross-attention. Token rates differ, so positions are mapped onto a shared timeline and temporal RoPE is applied to cross-modal queries and keys; neither stream is resampled.
Each cross-attention head is scaled by a sigmoid gate that depends on both the target hidden state and the attention output. A direction mask picks A2V, V2A or Joint per sample; the gate only scales an already-active path. In directional modes the target stream is noisier than the conditioner, and stop-gradient is applied to the conditioner keys and values so the target loss cannot rewrite the other backbone.
Clips are cut at scene boundaries and filtered with Q-Align, Audiobox Aesthetics, Synchformer and SyncNet, then captioned jointly by Qwen3-Omni. Speech is 45.0% of the data, event sounds 33.4%. Training runs in three stages: LoRA warmup on the latter half, full-parameter pretraining, then high-quality finetuning. Reinforcement learning routes video, audio and cross-modal advantages to the matching streams. The 2K refiner starts as a bidirectional multi-step teacher, becomes an autoregressive multi-step model, and is distilled with DMD into a student that does one denoising evaluation per temporal chunk. Audio is left untouched in this stage.
On Verse-Bench the 7B model is competitive with open research baselines and mixed against larger open-weight systems.
| Method | Params | VQ↑ | LSE-C↑ | DeSync↓ |
| NAVA | 6.3B | 0.6116 | 7.7261 | 0.2342 |
| Ovi | 10B | 0.6542 | 7.3095 | 0.4730 |
| Ours | 7B | 0.6568 | 7.8018 | 0.1902 |
| Ours (RL) | 7B | 0.6573 | 7.8361 | 0.1351 |
| Ours (Refiner) | 7B | 0.6930 | 7.6979 | 0.1731 |
| LTX-2.3 | 22B | 0.6285 | 7.6768 | 0.2412 |
| MiniMax-H3 | 33B | 0.6429 | 8.7354 | 0.2708 |
RL cuts DeSync from 0.1902 to 0.1351. The refiner lifts VQ to 0.6930 and slightly hurts sync. In a blind study, video-quality win rates against Ovi, UniAVGen, NAVA and DaVinci sit between 61.7% and 73.7%. Against Wan2.7, Kling v3 and MiniMax-H3, audio quality and AV alignment both lose more than they win.
This is the smallest disclosed open-weight native joint audio-video backbone with a local 2K path. Anyone working on joint modeling, sync rewards or video super-resolution distillation can start from a 7B checkpoint instead of a hosted API. It does not beat 22B/33B systems on audio aesthetics or cross-modal semantics. Treat it as a reproducible research base, not a leaderboard winner.
The 7B model trails LTX-2.3 and MiniMax-H3 on Content Enjoyment, Content Usefulness, Production Quality and ImageBind, and trails MiniMax-H3 on lip sync (8.7354). The human study under-samples highly dynamic, compositionally complex scenes. Refinement raises DeSync from 0.1351 to 0.1731, so resolution and sync do not improve together. RL is trained on only 1,000 first-frame/prompt pairs. The conclusion still describes RL and 2K refinement as designs with validation requirements, which sits awkwardly next to the reported tables; trust the tables.