CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer
Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, Da Yin, Yuxuan Zhang, Weihan Wang, Yean Cheng, Bin Xu, Xiaotao Gu, Yuxiao Dong, Jie Tang
cs.CV
2024-08-12
Tsinghua and Zhipu open-source CogVideoX (2B/5B) with a 3D causal VAE and expert transformer: 10s at 768×1360, five of seven auto metrics, 2.74 vs Kling 2.17 in human scoring.
After Sora put diffusion transformers on the map, open video models were still short, timid on motion, and weak on long prompts. The first CogVideo was an autoregressive transformer. The next wave of diffusion systems reused a 2D image VAE and split spatial and temporal attention. Flicker, broken large motion, and shallow text-video fusion follow from that split.
Tsinghua and Zhipu AI pack the alternative into CogVideoX: a 3D causal VAE that compresses space and time together, an expert transformer that puts text and video on one sequence, and a filter-plus-recaption data pipeline. The paper is at ICLR 2025. The 2B and 5B weights, the VAE, and the captioner are public. Maximum output is 10 seconds at 16 fps and 768×1360.
The 3D causal VAE maps pixels to latents at 8×8×4 (8× spatial, 4× temporal). Convolutions are causal in time: all padding sits at the start of the sequence, so future frames cannot leak backward. Pretraining uses variant B with 16 latent channels. Against SDXL's 2D VAE, adjacent-frame L1 flicker drops from 93.2 to 86.3 and PSNR moves from 28.4 to 28.7. Pushing compression to 16×16×8, even with 128 channels, is hard to converge. Long clips use temporal context parallel: each rank ships only a kernel-size-minus-one slice to the next rank.
On the expert transformer, T5 text embeddings and patchified video latents are concatenated on the sequence axis. The two modalities live on different numeric scales, so each gets its own Expert Adaptive LayerNorm, modulated by the diffusion timestep. Attention is 3D full attention, not 2D-then-1D. The paper's picture is concrete: under split attention, a person's head in frame i+1 cannot attend directly to the head in frame i, so identity has to travel through background patches. The 5B model uses 3D-RoPE (x/y/t take 3/8, 3/8, 2/8 of the channels); the 2B model keeps sinusoidal encodings.
Images are trained as one-frame videos. Multi-Resolution Frame Pack stuffs mixed durations and resolutions into one batch, so short clips are not dropped and long clips are not always truncated. Resolution climbs from a 256 short side to 512 then 768. Explicit Uniform Sampling slices the diffusion timestep range across data-parallel ranks so each rank samples inside its own bin; the loss jumps less. After filtering, about 35 million single-shot clips remain, averaging six seconds, plus two billion aesthetic-filtered images. Six Video-LLaMA classifiers cut recuts, low quality, talking-head lectures, text-heavy frames, and screen recordings. Captions start from Panda70M shorts, CogVLM dense frame descriptions, and GPT-4 summaries, then distill into LLaMA2 and finally CogVLM2-Caption.
The 2B model has 30 layers and hidden size 1920; the 5B has 42 layers and 3072. Diffusion uses v-prediction and zero SNR. A last fine-tune on about 20% higher-quality data strips subtitles and watermarks.
On WebVid validation at 256×256 and 17 frames, the VAE records flicker 85.5 and PSNR 29.1, ahead of Open-Sora (92.4 / 28.5) and Open-Sora-Plan (90.2 / 27.6).
Generation is scored with the VBench axes that track human taste, plus Dynamic Quality and GPT4o-MTScore:
| Model | Human action | Scene | Dynamic degree | Multi-object | Appearance | Dynamic Quality | GPT4o-MT |
| LaVie-2 | 96.4 | 49.59 | 31.11 | 64.88 | 25.09 | n/a | 2.46 |
| OpenSora V1.2 | 85.8 | 42.47 | 47.22 | 58.41 | 23.89 | 63.7 | 2.52 |
| CogVideoX-2B | 96.6 | 55.35 | 66.39 | 57.68 | 24.37 | 57.7 | 3.09 |
| CogVideoX-5B | 96.8 | 55.44 | 62.22 | 70.95 | 24.44 | 69.5 | 3.36 |
The 5B model leads five of seven metrics. Scene at 55.44 sits just under T2V-Turbo's 55.58. Appearance style trails VideoCrafter-2.0 at 25.13. Dynamic degree is higher on the 2B model (66.39 vs 62.22).
Human scoring against Kling (July 2024), each axis 0 / 0.5 / 1: sensory 0.722 vs 0.638, instruction following 0.495 vs 0.367, physics 0.667 vs 0.561, cover 0.712 vs 0.668, total 2.74 vs 2.17.
Architecture ablations: Expert AdaLN beats a same-parameter MMDiT on FVD; 2D+1D attention starts with worse FVD and is more likely to collapse. On an H800 with 50 steps, 5B needs 113 s and 26 GB for 480×720 at 6 s, and 500 s and 76 GB for 768×1360 at 5 s.
This is a reproducible open-source line from 2024-2025: 3D VAE, 3D attention, recaptioned data, and public weights. For anyone training a video DiT, the engineering bets are more useful than the leaderboard. Large motion should not rely on split spatiotemporal attention. Text and vision should not share one AdaLN. Caption quality may be worth more than another half-layer of architecture.
The 5B model beat then-current Kling in that human protocol. That comparison is a historical marker now.
The paper has no standalone limitations section. High-quality fine-tuning removes captions and watermarks; the authors note a slight drop in semantic skill. 3D full attention costs 9.60 s per forward at 768×1360 versus 4.17 s for 2D+1D, so consistency is paid in compute. Training clips are single-shot and about six seconds; multi-shot narrative is not what this data teaches. The human study only compares against Kling from mid-2024, and the main text omits rater count. Appearance style and scene are not uniformly leading. 16 fps and a 10-second cap look ordinary now.