VideoGen-Agent: Reinforcing Video Generation Agents
Binxu Li, Haoyi Duan, Yuhui Zhang, Yaohui Zhang, Zihao Lin, Kaituo Feng, Suozhi Huang, Xiangyi Li, Yu Li, Chunyuan Li, Shilong Liu, Mengdi Wang
cs.CV
2026-09-22
VideoGen-Agent learns retrieval, simulation, and verification tools. On VABench it lifts the base T2V from 56.5 to 75.6; swapping generators without retraining reaches 86.1.
Diffusion video models already look sharp and temporally coherent. Prompts that need specialized knowledge, a named identity, physical motion, or ordered events still drop steps, swap faces, or ignore gravity. Most agent pipelines are handwritten for one objective. Putting retrieval, simulation, generation, and checks behind one policy, chosen from the prompt and intermediate observations, needs multitask learning.
VideoGen-Agent trains one shared multimodal policy across six tasks: procedural knowledge, single-entity identity, multi-entity identity, physics, compositional scenes, and multi-shot structure.
The policy is Qwen3-VL-8B-Instruct in a reason-act-observe loop. Tools come in three groups. Augmentation: text search, image search, code simulation. Generation: T2V, I2V, multi-reference R2V, motion-conditioned M2V. Verification: object detection and depth. Default workflows point each task in a direction, e.g. search-then-T2V for procedures, last-frame conditioning for multi-shot.
Claude Opus 4.7 writes category-balanced prompts. Teacher agents (Gemini 3.1 Pro or Claude Opus 4.7) produce 24k trajectories: 16k for SFT, 8k held for RL. Failed actions are masked from the loss but kept in context. RL is GRPO with a hybrid reward: format 0.1, category-specific VLM video score 0.5, tool-use 0.4. Advantages are group-relative, then renormalized per task category. Generation tools are Toolset 1 during training; evaluation may swap in stronger Toolset 2 with the policy frozen.
VABench: 100 held-out prompts per category, 600 total, scored 0–100 by Gemini 3.1 Pro with the same rubrics used in training.
| System | Overall |
| Seedance 1.0 (base T2V) | 56.5 |
| Seedance 2.0 (strongest standalone) | 73.2 |
| Agent + Toolset 1 | 75.6 |
| Agent + Toolset 2 | 86.1 |
Toolset 1 is +19.1 over the base generator. Toolset 2 leads every category; multi-entity jumps from 65.3 to 86.7 and multi-shot from 81.7 to 94.6. Versus Seedance 2.0 the gaps are +14.1 procedural, +17.7 multi-entity, +25.7 multi-shot, and only +2.9 on composition (already 87.8). Four human raters prefer Toolset 2 in 84.3% of 100 pairwise comparisons.
Ablations on Toolset 1: prompt rewrite 57.3, zero-shot tools 59.5, SFT 69.2, full RL 75.6. Dropping the tool reward falls to 71.2; dropping the VLM reward to 73.3. Single-task RL scores 76.3, a hair above the shared 75.6. Mid-training, the loop can call π0.5 plus Ctrl-World for action-to-video. Unseen mixes such as a named character performing a specialized move trigger both text and image search.
Video generation often fails because the prompt needs external evidence: a procedure, a face, a physical trajectory, a shot list. Packaging those as tools and teaching an 8B policy when to call them adds 19 points without changing the base generator. Upgrading generators without retraining the agent shows the policy learned the interface, not one checkpoint. Composition gains little because the standalone model is already strong. Physics still sits at 69.2 even with Toolset 2, so simulation conditioning has not solved that class.
The judge and some teachers are Gemini 3.1 Pro, so the rubric is self-consistent and externally uncalibrated. Prompts are Claude-written and only spot-checked by humans, a synthetic distribution. Default workflows are written into the system prompt; the main generalization evidence is one spliced task. Generator quality and latency still cap the loop. Verification covers presence and spatial layout, not physics or identity drift. Single-task training is slightly stronger. Wall-clock end-to-end time is unreported.