AVA-Encoder: Towards Agent-Native Video Representation Learning
Chuyue Li, Jinpeng Yu, Haozhe Wang, Tian Xueyun, Zhijing Zhang, Bingnan Li, Shuqi Gu, Kan Ren, Jiaming Liu, Ruihua Hua
cs.CV, cs.CL
2026-08-13
AVA-Encoder encodes films into a knowledge graph, reconstructs them with a fixed decoder, and self-improves from reconstruction error, beating the best baseline by 20.7 points.
Video-creation agents can already write stories, design keyframes, and generate clips, but they're still far from producing a complete, release-quality film, largely because their base models lack the planning ability to coordinate scripts, characters, shots, and audiovisual elements together. The best teacher for that kind of planning is the huge body of professionally directed film, rich in screenwriting, character design, camerawork, pacing, and audiovisual coordination, that human filmmakers study closely. Agents can't use the same films that way: film is tightly woven multimodal content, while agents operate through structured representations like text, code, and graphs, and the two don't line up. Low-level visual representations, pixels, latents, carry plenty of information but agents can't read or edit them; text captions are readable but compress a film into a single linear description, losing entity relationships, event structure, and cross-modal dependencies; existing video knowledge graphs are mostly built for retrieval and QA, storing sparse semantic facts rather than the dense detail needed to actually reconstruct a shot.
The paper proposes a film-creation knowledge-graph representation: a Story-Event-Shot narrative hierarchy, with Character, Scene, Object, Style, Camera, and Audio state nodes attached to each Shot, all storing structured text, while generated keyframes, audio, and video live only in a separate asset layer connected by typed edges, so following one broken edge lets you fix everything downstream of it. Around this representation, AVA-Encoder is a self-evolving encode-decode framework:
On a ground-truth-anchored reconstruction benchmark, AVA-Encoder reaches 49.0% overall reconstruction accuracy versus 28.3% for the strongest external baseline, soap2soap, a 20.7-point gap, and leads across all four comparison directions: direct video comparison, direct keyframe comparison, video back-captioning, and keyframe back-captioning (ahead by 21.1, 34.2, 13.9, and 11.6 points respectively). Ablations trace where the gain comes from: three-level hierarchical understanding beats single-level by 18.3 points (27.5 to 45.8, 66.5% relative); in a controlled comparison with the inner loop disabled, the pseudo-trained policy (45.8) beats the human-tuned policy (44.4) while using a system prompt of just 8,052 tokens versus 31,336, a 74.3% reduction; the two loops each contribute independently and together add 6.6 points (15.6% relative) over using neither; removing the anti-degradation and anti-forgetting gates drops accuracy from 49.0 to 43.5, showing the gates are actively catching regressions, not just insurance that never triggers. The graph's editability is also demonstrated directly: editing one character's registry entry lets an LLM follow the graph's dependencies to update that character's appearance, actions, and dialogue across every affected shot, while leaving unrelated characters and scenes untouched. Feeding the generated knowledge graph as plain text context into four existing downstream video-generation frameworks (MovieAgent, FilmAgent, Anim-Director, VideoStudio), with no framework-specific adaptation, improved every one of them.
The core contribution here isn't a better video-generation model, it's a reusable intermediate representation that translates a film into text: once a movie has this knowledge graph, an agent can understand what it depicts and how it was shot, and edit it like a document instead of regenerating whole sequences. For creative agents, this is the first real path to turning professionally made films into learnable, searchable, reusable training material, rather than pixel streams a human can watch but a machine can't operate on. Fixing the decoder and treating reconstruction error as the optimization signal also gives a concrete way to measure how much information a representation preserves, instead of inferring representation quality indirectly through downstream task performance.
The paper doesn't have a dedicated limitations section, but the design reveals its boundaries. Reconstruction quality depends heavily on the fixed decoder's own capability (text-to-image plus image-to-video); even a perfectly accurate knowledge graph can't score well if the decoder can't render the corresponding scene, and the paper doesn't discuss how swapping in a stronger or weaker decoder would change the results. The 18 evaluation clips span animation, AI-directed shorts, and classic cinema, but the sample is modest, and pseudo-training uses only six videos, so it's unclear whether results hold at larger scale. Using a VLM for fact-based QA scoring is more fine-grained than a single holistic judgment, but any systematic bias in that judge, missing certain visual details consistently, propagates directly into both the reconstruction reward and the final evaluation; the paper applies a pairwise consistency check (must be preferred in both presentation orders) but doesn't separately quantify the judge model's own error rate.