SemComp-Bench: Benchmarking Semantic Task Completion in Video Generation
Keyu Tu, Zhuowei Chen, Mengqi Huang, Yuxin Wang, Jiahao Zhu, Zhendong Mao, Yongdong Zhang
cs.CV, cs.AI
2026-08-18
SemComp-Bench scores outcome achievement and generation reliability with VLM questions on 1,273 clips. HunyuanVideo I2V leads OA at 37.8%; Seedance 2.0 leads GR at 91.8%.
Video generators now look sharp and temporally smooth. Benchmarks followed that progress: fidelity, identity, motion, even crude physics. What they rarely test is the job a user actually assigns: given a reference image and an instruction, produce a clip whose completed state is both achieved and semantically tied to that image.
The paper’s running example is a banknote with the prompt “fold it into a turtle.” Intermediate folding can be omitted. The finished origami still has to be that banknote, not a generic turtle. Pixel-level appearance lock is not required. Existing suites let a model keep the look, mime a process, and miss the outcome.
The task is Semantic Task Completion Video Generation. Success is outcome achievement plus task-relevant semantic grounding. Full procedure and conventional appearance consistency are out of scope.
SemComp-Data is built from roughly 20K Koala-36M videos. A four-stage pipeline turns them into image–text–video triplets. Title keywords drop talk-heavy videos. A VLM bins remaining clips into six domains. Category-specific definitions of reference and outcome states are localized as timestamps, then checked with a blind “which frame is the outcome?” question. A shot-aware crop around the outcome yields clips of about 4.03 seconds. Brief instructions stay under 30 words; detailed ones add grounding constraints. Reference and outcome always come from the same source video, so the task is visually feasible. The full set has 1,273 instances; SemComp-Core holds 60, ten per domain.
SemComp-Bench asks Doubao-Seed-1.8 structured binary questions with visual evidence. OA is conjunctive over four gates: outcome realization, semantic grounding, grounded-entity consistency, and global visual continuity. GR averages five reliability gates: physical plausibility, clarity, artifact-free rendering, within-scene spatiotemporal coherence, and text/UI integrity. Each clip is scored from 27 uniformly sampled frames, three independent VLM calls averaged.
Under detailed-instruction I2V, HunyuanVideo-1.5-720P leads OA at 37.8%, with Wan2.2-I2V-A14B at 28.3%. Seedance 2.0 is strong on realization (0.839) and grounding (0.744), then collapses on entity consistency (0.444), so joint OA is 20.0%. Phantom-1.3B sits at 3.9% OA. The best score is still under 40%.
| Model | OA Score | GR Score |
| HunyuanVideo-1.5 I2V | 37.8% | 79.1% |
| Wan2.2-I2V-A14B | 28.3% | 89.0% |
| Seedance 2.0 | 20.0% | 91.8% |
| Phantom-1.3B | 3.9% | 76.8% |
Seedance 2.0 leads GR at 91.8%; Wan2.2-A14B is the best open model at 89.0%. Within-scene spatiotemporal coherence is the shared bottleneck, with pass rates from 0.328 to 0.739.
Conditioning is decisive. Across three families, I2V beats same-size T2V: HunyuanVideo falls from 37.8% OA (detailed I2V) to 4.4% (detailed T2V) and 1.7% (brief T2V). T2V can still “make an outcome”; it fails to keep the referenced entity and scene. Brief prompts, closer to user habit, are harder on fulfillment.
This is an outcome ruler, not another beauty contest. OA and GR disagree: Seedance is reliable and weak on completion; HunyuanVideo is the reverse. A pretty frame is not a finished task. Image conditioning is close to mandatory if identity has to survive. Whether training on SemComp-Data lifts OA is left untested.
The headline ranking uses 60 clips. Judging is a single closed VLM; OA run-to-run standard deviation reaches 4.41 points. Intermediate procedure is not scored, and the physics gate only flags obvious violations. Source videos are in-the-wild, not controlled. No generator was trained on this data, so the suite diagnoses, it does not close a training loop.