Evaluation Benchmark for Keyframe Video Generation
KlingTeam · hf · 2026-07-17
This work proposes KeyFrame-Compass, designed to systematically evaluate keyframe-conditioned video generation.
Benchmark Contents
The benchmark contains 386 samples, covering:
- 3 application domains
- 2 video structure types
- 2 prompt granularities
- 2 condition formats
- 4 keyframe densities
This allows for comparing model performance differences under various settings.
Evaluation Method
The authors break down "keyframe execution" into 6 dimensions:
- Appearance
- Fidelity
- Temporal order
- Localization
- Persistence
- Uniqueness
It also evaluates overall video quality, combining evidence-driven MLLM judgments with specialized perceptual models.
Main Findings
Across 9 representative video generation systems, the authors found:
- The stricter the keyframe adherence, the more video naturalness tends to suffer
- The denser the keyframe constraints, the worse the performance
- Many open-source models even fail to interpret storyboard-grid inputs as temporally ordered keyframe sequences
More from Multimodal
- TischEins, a large Midjourney SREF library, relaunches with new design after security outage — tisch_eins · 2026-10-26
- MOSS-Image-SFT 4-step distillation tops 16.5K downloads on Hugging Face — _akhaliq · 2026-10-26
- Claude Code can now edit videos: a 10-prompt workflow from raw footage to final cut — TawohAwa · 2026-10-26
- One-sentence 3D worlds turn cinematic with CineMap and TapNow — gizakdag · 2026-10-26
- No Camera Needed: Creator Builds an Entire World in CineMap and Directs It Into Cinema — gizakdag · 2026-10-26
- Kling vs. Seedance: the Chinese AI video model rivalry — DavidmComfort · 2026-10-26