CoEvoWhen: Policy-Tool Coevolution for Ultra-Long Video Temporal Grounding
Yiduo Jia, Muzhi Zhu, Jinchuan Shi, Hao Zhong, Yuling Xi, Ke Liu, Hao Chen
cs.CV
2026-10-01
CoEvoWhen freezes the VLM and evolves its skill from execution traces, rewriting policy docs and media-tool code: hour-long video grounding mIoU up 74.9%, visual tokens down 11%.
Video temporal grounding asks a model to find the start and end times of an event described in a plain-language query, inside an untrimmed video. Once videos stretch to tens of minutes or hours, it becomes a budget problem: the target can occupy a sliver of the timeline, and one case in the paper hunts an 11-second scene inside a 100-minute video. Coarse sampling misses brief events; repeated close inspection burns visual tokens. Balancing long-range evidence search against fine-grained local understanding under a fixed visual budget is the core tension.
Agentic systems such as VideoAgent, VideoTree, and DVD already replaced one-shot prediction with multi-step search and tool calls, but their observation mechanisms and media-processing capabilities stay hand-designed. The paper starts from the observation that tools and policies are interdependent: tools determine what the model actually sees, policies decide when and how to look. Refining policies alone hits the ceiling of existing tools; adding tools alone does not mean the agent uses them well.
CoEvoWhen externalizes competence as an editable skill with two parts: policy documents (a SKILL.md plus notes on planning, tool selection, and tool usage) and a set of executable media tools, each shipping a description, an interface spec, and source code. The VLM's weights stay frozen throughout.
The loop runs like this:
The load-bearing design choice is the division of labor between image and video observation. Contact sheets, meaning timestamped thumbnail grids, cover tens of minutes in a single image observation and suit global candidate comparison; video clips preserve local temporal continuity for judging action order, state changes, and boundaries. Coevolution leaves this split unwritten and lets trajectory feedback shape both sides. The converged skill reads as image-based global search, candidate refinement, then selective video verification: on VUE-LVTR, per-query video tokens fall from 40.6k to 0.5k, down 98.7%, with the budget shifted to images. At inference the skill is fixed, the policy enters the system prompt, tools register as callable functions, and the VLM decides on its own for up to 24 rounds, with no separate stronger planner.
Main experiments use Qwen3.5-27B, with base and evolved skills compared under identical protocols alongside seven VLM and agent baselines.
| Benchmark | Metric | Base Skill | Evolved Skill | Change |
| VUE-LVTR held-out (307 queries, mean 50.9 min) | IoU AUC | 0.4107 | 0.5137 | +25.1% |
| ExtremeWhenBench (2,273 queries, mean 75.8 min) | mIoU | 0.1507 | 0.2636 | +74.9% |
| ExtremeWhenBench | [email protected] | 0.1478 | 0.2785 | +88.4% |
| CoMET-Bench (multi-event, 1,599 queries) | Rejection-F1 | 66.13 | 77.06 | +10.93 |
| LVBench long-video QA (skill transferred as-is) | Accuracy | 45.45% | 54.10% | +8.65 pts |
Among external baselines on VUE-LVTR, GPT-5.6 Luna with 128 timestamped frames reaches 0.2510 IoU AUC, TimeLens-7B 0.2758, VideoMind-7B 0.2000; the evolved skill's 0.5137 leads all seven. Efficiency moves in the same direction: cumulative visual tokens per query drop from 202.6k to 141.9k on VUE-LVTR (−30.0%) and by 11.4% on ExtremeWhenBench, with model calls down 43.5% on VUE-LVTR. Table 1 also carries a quieter point: the base skill alone lifts the plain VLM from 0.1904 to 0.4107 IoU AUC, so turning grounding into multi-step search is the single biggest factor, and coevolution adds another +25.1% on top.
The ablations are thorough. Policy-only and tool-only variants both beat the base skill, but joint evolution exceeds them by 0.0243 and 0.0735 IoU AUC while using 25.5% and 40.6% fewer visual tokens; tool-only actually pushes per-query tokens above the base level, to 238.9k. Freezing policies costs more than freezing tools, which says strong observation capabilities pay off only when organized into an efficient search process. Image-only and video-only skills both improve with evolution, yet the coordinated two-modality skill beats both on every metric with fewer tokens. Skills evolved separately on Qwen3.5-9B and Qwen3.6-27B gain 99.2% and 24.7% mIoU on ExtremeWhenBench.
For teams building on long video, this is a recipe for turning task experience into an asset that never touches weights: a few Markdown files and some Python. Evolution needs one pass over 100 labeled queries, orders of magnitude below any fine-tuning run. The skill also transfers unchanged to long-video QA, adding 8.65 points on LVBench and 7.98 on LSDBench, evidence that the learned observation orchestration generalizes beyond one benchmark. Its honest position: scaffolding engineering, not a new model. The delta over Voyager- or ExpeL-style experience accumulation is that tool evolution is free-form code synthesis instead of predefined operations, and the task-level orchestrator sits inside the evolution loop.
The authors' own list: evolution uses 100 queries from a single source (VUE-TR-V2) and requires labeled intervals as feedback; only the visual modality is used, with no audio or transcripts; all observation is done by the main agent, and delegating to subagents is left to future work.
Points that hold up less well. The updater is Codex GPT-5.5 at xhigh, a proprietary model; the paper reports no cost for the evolution pass and no ablation on updater capability. The largest gain (25.1%) sits on a held-out set from the same distribution family as the evolution data, while the cross-dataset ExtremeWhenBench gain starts from a base of 0.15 mIoU, so most queries are still localized poorly. Gains shrink as the backbone strengthens, +24.7% on Qwen3.6-27B versus +99.2% on the 9B. On CoMET-Bench queries with 16 or more events, [email protected] stays below 0.06, leaving dense multi-event cases far from usable.