Evaluation Benchmark for Keyframe Video Generation
KlingTeam · hf · 2026-07-17
This work proposes KeyFrame-Compass, designed to systematically evaluate keyframe-conditioned video generation.
Benchmark Contents
The benchmark contains 386 samples, covering:
- 3 application domains
- 2 video structure types
- 2 prompt granularities
- 2 condition formats
- 4 keyframe densities
This allows for comparing model performance differences under various settings.
Evaluation Method
The authors break down "keyframe execution" into 6 dimensions:
- Appearance
- Fidelity
- Temporal order
- Localization
- Persistence
- Uniqueness
It also evaluates overall video quality, combining evidence-driven MLLM judgments with specialized perceptual models.
Main Findings
Across 9 representative video generation systems, the authors found:
- The stricter the keyframe adherence, the more video naturalness tends to suffer
- The denser the keyframe constraints, the worse the performance
- Many open-source models even fail to interpret storyboard-grid inputs as temporally ordered keyframe sequences
More from Multimodal
- Dev builds interactive 3D product experience with GPT-6 Astra + Hyper3D Rodin — nikola_mr64990 · 2026-09-11
- Using a finisher move on one mosquito with MiniMax H3 MAX — the bug survives — Hailuo_AI · 2026-09-11
- Skyfall GS Uses Flux to Refine Gaussian Splatting, Accepted at ECCV 2026 — ducha_aiki · 2026-09-11
- Lumara AI Film Festival Comes to NYC Oct 26, Top AI Filmmakers to Compete — 0xAllen_ · 2026-09-11
- Pterodactyl Detective: An AI-Generated Proof-of-Concept Trailer — PterodactylDetective · 2026-09-11
- Imperium Game Trailer Showcases AI Video Generation — keaslenyt · 2026-09-11