Behind Video Generation: Observations and Insights on Multimodal Understanding from Kling
Kling Team, Dong Chen, Jialu Chen, Xinlong Chen, Yifan Dai, Haotian Dong, Binzong Geng, Yushuo Guan, Minghui Guo, Guanlin He, Guanxiao He, Jingyun Hua, Yanwei Huang, Yiyan Ji, Xueping Jia, Yizhuo Jia, Bozhou Li, Jiaze Li, Min Li, Shihao Li, Shujie Li, Shengyi Liao, Weihong Lin, Shiyuan Liu, Xinyu Liu, Siyang Mao, Xiaoru Qu, Dianmo Sheng, Yang Shi, Jiafu Tang, Longxiang Tang, Yuqi Tang, Chengzhuo Tong, Kuo Wang, Qixun Wang, Rui Wang, Tianduo Wang, Yuran Wang, Zekun Wang, Zihao Wang, Boyang Xia, Yifan Xu, Liu Yang, Zhenlong Yuan, Bohan Zeng, Mujin Zhang, Yuanxing Zhang, Zhicheng Zhang, Zhuoran Zhang, Yuanhong Zheng, Kun Gai, Chengru Song, Pengfei Wan
cs.CV
2026-11-03
Kling details the understanding stack behind video generation: one MLLM specialized into captioners, prompt rewriters, and reward models. Kling-Captioner-397B tops VidCapBench.
By 2026, frontier video generators produce usable short-form audiovisual sequences in which subjects, camera language, dialogue, and music hold together. The Kling team's reading: rendering is no longer the bottleneck. Semantic control is, meaning the generator reliably executes who does what, how the camera moves, and which elements must stay untouched.
The report reviews the multimodal understanding stack behind Kling 1.0 through 4.0. Its thesis in one line: generation quality is the projection of multimodal understanding through the entire data, conditioning, alignment, and evaluation chain. A single generation failure is not a diagnosis. The same symptom can come from a caption that omits key supervision, a conditioning representation that drops temporal structure, a reward model that favors smooth but static motion, or an evaluator that never tests the requested event. Aggregate VQA scores do not cover camera language, left-right relations, or identity binding, so the stack has to be built module by module.
The foundation, Kling-MLLM, comes in a dense 27B variant and two MoE variants (35B-A3B, 397B-A17B), with LLM and vision backbones initialized from Qwen3.5-Qwen3.8 checkpoints, and it unifies image, video, audio, synchronized audio-video, and interleaved inputs in one autoregressive interface. The audio token rate is fixed, so elapsed time should be recoverable from token position; in practice it is not, and duration awareness and temporal grounding both suffer. The fix is start and end timestamps around each audio span plus absolute anchors every five seconds, with synchronized audio and video sharing one timestamped timeline. Training follows a four-stage curriculum: first the audio tower alone with vision and LLM frozen, then full joint training, then long sequences for temporal capacity, then frozen modality towers with only the LLM updated on the highest-quality data. Merging the Stage 1 and Stage 4 checkpoints beats either checkpoint alone.
The foundation is then specialized into the chain's modules. Captioning is treated as a semantic contract. A DiT trains on far fewer samples than a language model and the gap cannot be closed by scaling, so textual description density per second of video becomes the central measure of data efficiency. The contract mandates six information families, from subjects and action through space, cinematography, shot timing, visible and audible text, to aesthetics; a multi-expert pipeline produces the supervision and is distilled into a single-pass student model. Reference-conditioned generation extends this into RGC: each caption states explicitly which reference binds to which entity, attribute, and time span, everything the references cannot supply is written out, and whatever a reference already governs is not re-described in text. The cheapest data trick is relational supervision: any two sufficiently similar videos form a training pair, one as reference and one as target, with no real edit required. Prompt enhancement translates user requests into this caption distribution, with training pairs built backward from captions of compelling videos; at inference, an agentic stage repairs story contradictions and retrieves five compelling captions before the rewriter produces the final condition. Two further conclusions: a reliable text encoder comes from perceptual training rather than stronger language ability, with candidates screened from text alone before end-to-end training, and reward modeling splits into a general model for explainable, localized critiques plus a decision model for fast multidimensional scoring.
The foundation comparison is not one-sided; the 397B advantage concentrates in temporal perception, audio-video captioning, and camera understanding.
| Setting | Metric | Kling-MLLM-397B | Best external baseline |
| video-SALMONN (audio-video captioning) | total error, lower is better | 37.07 | 39.13 (Qwen3.5-Omni-Plus) |
| Camera Motion VQA | accuracy | 54.72 | 37.70 (Qwen3.5-Omni-Plus) |
| DSI-Bench (temporal understanding) | accuracy | 65.52 | 63.90 (Qwen3-Omni) |
Qwen3.5-Omni-Plus still leads MVBench at 80.37, and Doubao2.0-Lite (thinking) takes LongVideoBench (75.17) and MME-VideoOCR (84.65). Image and speech-recognition columns trade wins across systems. On VSI-Bench, Kling-MLLM-35B-A3B scores 75.07 against 64.16 for the best external baseline, Qwen3-VL.
The specialized captioners post the strongest numbers in the report:
| Model | VidCapBench accuracy / precision / coverage |
| Gemini-4-Argon | 16.20 / 51.66 / 85.63 |
| Doubao2.1-Lite (thinking) | 20.23 / 56.44 / 94.15 |
| Kling-Captioner-397B | 21.70 / 60.51 / 96.05 |
Kling-Captioner also leads UGC-VideoCap and the internal motion, camera, and sound evaluations. On reference-grounded captioning, Kling-RGC-397B averages 78.89 on ViDiC against 77.18 for Gemini-4-Argon, and leads MRVBench keypoint coverage (95.18), VQA (76.34), and reference-binding precision (94.85). Gemini keeps Ref-Tag-R (97.23 vs 95.28), Subj-F1 (90.37 vs 85.24), and MRVScore (91.22 vs 90.31).
For generation teams, the value is the methodology rather than weights: description density as the data-efficiency metric, the three-part RGC contract (scoped binding, residual sufficiency, authority consistency), training pairs from arbitrary similar videos, and backward-constructed PE data are all portable engineering decisions. For anyone building understanding models, it sketches a productization path: one foundation specialized into five roles (captioner, PE, encoder, reward model, evaluator) at different accuracy-latency points, with evaluation decomposed per task and cheap text-only screening filtering candidates before expensive end-to-end training.
The authors' own list: highly dynamic content forces a trade-off between temporal and spatial fidelity and context capacity; long-form video pits caption length against density, and referential consistency over long horizons is unsolved; audiovisual alignment and speaker attribution need stronger integration; and real user requests arrive without a complete binding structure, so translating underspecified requests into the scoped caption distribution used at training time remains open. Two concerns from this read: most conclusions arrive as retrospective narrative without controlled ablations in the visible sections, and the claim that higher description density buys better generation is stated as a principle with no number attached. The evaluations are self-run under LMMs-Eval, some baselines lack directly comparable official scores, and the foundation comparison is mixed, so the framing carries more weight than the scoreboards.