TimelineBench: Best of 16 AI Agents Passes Just 26.8% of 56 Real Video-Editing Tasks
ycombinator · x · 2026-10-02
A new arXiv paper introduces Timeline-Bench, a benchmark of 56 real video-editing tasks where agents must turn raw production footage into a finished cut — selecting dialogue takes, shaping interviews into stories, or cutting commercials from product shots, voiceovers and graphics. Each task ships with a brief, source assets, a container, and tests; quality checks are calibrated on 2,582 blind judgments from 43 video editors.
Key findings:
- 16 agents pairing frontier models with harnesses like Codex, Claude Code and OpenCode were evaluated
- The best, GPT-6 Astra in Codex with curated editorial guidance, resolves only 15 of 56 tasks (26.8%); the average agent resolves 14.0%
- Human editors prefer the reference edit in 83.5% of judgments
- Of 771 failed runs, 562 fail only the quality test: agents perceive footage via stills and transcripts and check renders for defects, not craft
Tasks, verifier and per-run results are released openly.
More from coding & agent
- Generic API-catalog MCP server vs one tool per job: a context-bloat experiment — MandeepKaur9407 · 2026-11-16
- Berkeley paper: LLMs answer with stale values even when the update is in context — ombre-ai · 2026-11-16
- skills-gate: one gate skill fronts 148 agent skills at just 389 tokens — Distinct-Pie2389 · 2026-11-16
- Open-weight Karpo-Decision-31B tops Decision Index at 73.9, one forward pass per answer — CodeByPoonam · 2026-11-16
- shadcn boosts @uisdk: one-shot UI rebuild that's instantly E2E-testable by AI — shadcn · 2026-11-16
- AI agent deletes 43,599 lines of OpenClaw test code in one PR — vincent_koc · 2026-11-16