Visual prompt engineering for video models
Robert Geirhos, Yuxuan Li, Thaddäus Wiedemer, Neha Kalibhat, Zi Wang, Mani Malek, Oyvind Tafjord, Kevin Swersky, Been Kim, Priyank Jaini
cs.CV, cs.AI
2026-07-28
DeepMind shows that editing a video-reasoning task's input image from an abstract sketch to photorealistic systematically lifts performance; on Veo 3.1 a physics task goes from 41.3% to 59.3%, beating text-prompt edits and test-time sampling.
Text-prompt engineering is standard practice for language models. Video generation models are becoming foundation models for visual reasoning (solving mazes, logic puzzles, physics), and they take both a text prompt and a "visual prompt" (usually the first frame of the generated video). Oddly, the text input has been studied endlessly while the visual input is treated as fixed. This work fills the gap: can we improve a video model's reasoning by engineering the task image, the way we engineer the text prompt?
They call it visual prompt engineering (VIPE), in three steps:
Automation comes in two modes: freeform ideation, where a VLM (Gemini 3.1 Pro) proposes edits open-loop, and ACE (Atomic Concept Editing), which edits one concept at a time in a tree search and iterates closed-loop on autorater feedback.
On VPCT, a physics-reasoning task (predict which bucket a ball lands in), turning the sketch photorealistic lifts every model: Veo 3.1 from 41.3% to 59.3%, Omni Flash from 56.3% to 67.5%, Wan2.2 from near-chance to clearly above chance. The key contrast: test-time self-consistency voting only takes Veo 3.1 from 41.3% to 50.0% (20 samples), while a single VIPE sample reaches 59.3%, and combining the two reaches 68.0%. A VIPE variant costs one-eighth of a video generation. Across six tasks and 18,160 videos, some task variants cut the error rate by over 75%, and ACE reaches a 100% error reduction on Sort 3 Numbers. Compared head-to-head with text-prompt engineering, Sort 3 Numbers goes from 4% to 86% (best text variant) and 76% (best image variant), with image variants averaging larger gains on several tasks.
The insight matters for both evaluation and practice. For evaluation, it exposes a realism bias: video models do poorly on abstract, synthetic tasks, but that may not reflect a lack of ability, just that they were pushed out of distribution, so the many abstract sketch benchmarks likely underestimate models. The authors' explanation is that video models train mostly on realistic video, so feeding them abstract sketches forces reasoning on unfamiliar ground, where scene consistency breaks down (objects randomly appear, disappear, morph) and reasoning fails; each step toward realism raises consistency (human-rated scene consistency rises from 0% for synthetic to 59% for fully realistic). For practice, VIPE is a cheap (one-eighth of a video), plug-in test-time lift.
The authors state the main limit themselves: variant quality depends heavily on the image-editing model, and if the editor changes the task or introduces artifacts, the downstream video model cannot solve the original task; they mitigate this with the Filter step, which may become unnecessary as editors improve. A few open questions remain. VIPE is validated mostly on structured, clearly describable reasoning tasks; its effect on open, ambiguous real-world visual tasks is unclear. The realism bias is itself an unwanted shortcut (an ideal model should solve a task in any form), so using it as a tool is a stopgap that points to a training-distribution defect. Automated VIPE also depends on the Gemini and Nano Banana stack, so reproducibility is tied to those closed models.