Timeline-Bench: Evaluating Agents on Realistic Video-Editing Tasks, from Raw Footage to Final Cut
Gunin Gupta, Nirmit Arora, Pavan Kalyan Tankala
cs.CV, cs.AI, cs.MM
2026-09-28
Timeline-Bench gives 16 frontier agents 56 real video-editing jobs; the best, GPT-6 Astra in Codex, resolves 26.8%, and 73% of failed runs pass every test except quality.
Agent benchmarks usually grade an end state: SWE-bench and Terminal-Bench pass or fail on tests, which works when a task has one acceptable outcome. A video edit does not. Many different cuts satisfy the same brief, so compliance is easy to check and quality is hard. Professional benchmarks like GDPval bring in expert judges but carry few media tasks, so editorial capability stays invisible in their aggregate scores.
Timeline-Bench makes this concrete with 56 real editing assignments: picking takes from interview footage and shaping them into a story, cutting narrative scenes from a script, assembling commercials from product shots, voiceovers and graphics. Each task hands the agent a brief, raw assets, a Docker container and a time limit, and only the finished video counts. Source material totals about 33 hours, drawn from purchased EditStock packs, public Cinestudy projects and 30 newly commissioned productions, with delivery windows from 25 to 315 seconds.
A task is a brief, source assets, a Docker image, a test suite and a 300-minute limit on a dedicated 32-vCPU, 256 GB machine. Only the delivered video is scored; the agent picks its own workflow.
The tests come in four layers:
Anti-gaming is thorough. Reference edits never enter the container (MD5-checked against all 5,003 input files), each task ships a deliberately crude but compliant mechanical oracle to validate the delivery tests, and single-defect controls (muted audio, blacked-out opening, reordered sections) confirm the tests reject bad edits.
Sixteen agents: ten models in the open-source OpenCode harness, GPT-6 Astra and GPT-5.6 Sol in Codex CLI, two Claude models in Claude Code, plus two variants, GPT-6 Astra in Codex with 1,909 words of generic editing advice and a computer-use version driving DaVinci Resolve. One run per task, 896 runs total.
| Agent | Resolution | Human win-or-tie |
| GPT-6 Astra (Codex + guidance) | 26.8% (15/56) | 24.4% |
| Claude Opus 5 (Claude Code) | 23.2% | 23.2% |
| GPT-6 Astra (Codex / OpenCode) | 21.4% | 23.8% / 22.3% |
| GPT-6 Astra (computer use) | 3.6% | 8.2% |
| All 896 runs | 14.0% | 16.5% |
Human editors prefer the reference edit in 83.5% of judgments, the agent edit in 11.1%, and call it a tie in 5.5%. With one run per task, each agent's 95% CI spans about ±11 points, so the leaders are statistically indistinguishable. No agent resolves 35 of the 56 tasks, against roughly 5 expected if successes fell at random.
The quality test is where agents lose: 562 of 771 unresolved runs (73%) pass every other test. The diagnosis is specific. Agents see footage as stills, with 83% of source reads opening frames or contact sheets, and GPT agents in Codex even print audio excerpts as base64 text. They cut at 0.72 times the reference pace and hold their longest static shot 1.5 times as long. Self-checks are nearly universal (86-100% of runs for 14 of 15 code agents) but target technical defects: 5.5% of the problems agents report concern pacing, story or shot choice, against 66% of human editors' notes. And 93% of runs end with the agent claiming full success, including 95.5% of edits that fail a test.
This turns "can agents deliver creative work" into a reproducible measurement: tasks, verifier and per-run results are all released (Apache-2.0), and the quality test tracks the human win-or-tie rate at ρ=0.93, so new submissions can be scored without convening a human panel. The diagnosis is more useful than the ranking. Whatever a brief can state, agents already do reliably, with 857 of 863 delivered videos meeting the delivery spec; the gaps are native audiovisual perception and judgment of their own work. Two contrasts stand out. The same model resolves 21.4% of tasks writing code but only 3.6% clicking through editing software, partly because it cannot hear and misses 41.5% of required spoken lines. And swapping harnesses never reaches significance in matched contrasts, while the computer-use deficit does (p<0.001). Per-run cost runs from $0.13 to $41.27; the champion configuration is an estimated $9.76.
The authors list three: one run per task, so intervals omit run-to-run variance; the quality test is calibrated on the same human votes it is validated against; and it holds per agent, not per edit, agreeing with the human majority at only κ=0.15 on single edits. Reference edits are professional work, not certified ground truth.
A few more from the reading. Single human judgments are noisy (Krippendorff's α=0.10), and it takes three editors per pair to lift per-agent reliability to 0.84. The curated guidance lifts GPT-6 Astra from 21.4% to 26.8%, but the matched contrast is not significant (p=1.00), and the gap to the runner-up sits inside the noise. The three quality judges are also models under evaluation; the cross-laboratory panel mitigates this, but no per-judge systematic bias is reported. Source media are gated to approved research teams, so reproducing the eval requires an access request.