VideoLoop: Looped Working Memory Against Semantic Thrashing in Long-Form Video Agents
Jinfa Huang, Jianming Xu, Jingyang Lin, Zhengyuan Yang, Jiebo Luo
cs.CV
2026-09-30
VideoLoop rewrites bounded working memory from a filesystem after each step. Four LVLMs gain 4.2 points on VideoMME (long); Gemini 3.1 Pro hits 88.3%.
Long-form video QA has to pull sparse evidence from thousands of frames over tens of minutes. The usual fix is an agent: look at a slice, append the observation to working memory, decide where to look next. VideoAgent, DVD, VideoSeek, and VideoARM all do this, and iterative search does beat stuffing the whole video into one pass.
The failure mode is append-only memory. As the log grows, attention on evidence the agent already found gets washed out. University of Rochester and Microsoft call this semantic thrashing, by analogy with OS thrashing: the machine stays busy while useful work collapses. A set-difference argument makes the constraint sharp. Appending can admit newly seen target evidence. It cannot delete noise that already entered, and it cannot stop ordered context from growing. Without a rewrite operator those two failures stay.
VideoLoop is two coupled loops. The outer loop watches the video. The inner loop maintains memory.
The outer agent runs in a sandbox with four tools: ANALYZE answers a self-generated subquestion on selected frames; TRANSCRIBE returns a timestamped transcript and caches it; EXECUTE runs arbitrary code and writes frames, captions, and scripts to a filesystem; ANSWER commits and stops. At each step the policy sees only the user question, the current bounded working memory, and a sliding window of the last k=8 thought-action-observation triples. Raw artifacts from older steps never enter the outer context.
The inner loop is a second LLM, the memory orchestrator. After every outer step it retrieves relevant artifacts from the unbounded filesystem and rewrites working memory under a BM=32K token cap. Memory is a six-section document: metadata, narrative understanding, timestamped evidence, temporal coverage, activity log, and open questions. Edits are UPDATE, APPEND, and DELETE. Syntax is per-section; the decision is joint.
Storage has three tiers: the bounded document the outer loop can see; a compressed action manifest that folds the oldest 10 records into one summary once 30 unsummarized entries pile up; and a lossless filesystem. A rewrite lowers the symmetric difference with the latent evidence set only if dropped noise plus recovered targets outweigh deleted targets plus new noise. That is a quality condition, not a guarantee. If the agent has not ANSWERed by N=50 steps, the policy is queried on the last memory. Memory starts empty. At most 6 keyframes sit in working memory. Inner and outer loops share the same backbone by default.
Benchmarks use official subtitles except VideoMMMU, which is transcribed with Whisper-large. Native Gemini 3 Flash and Gemini 3.1 Pro numbers are the authors' own reproductions.
| Setup | VideoMME (long) | VideoMMMU | LongVideoBench (long) |
| Gemini 3 Flash native | 80.7 | 83.6 | 67.9 |
| VideoLoop | 85.8 (+5.1) | 87.7 (+4.1) | 73.8 (+5.9) |
| Gemini 3.1 Pro native | 83.8 | 84.6 | 77.7 |
| VideoLoop | 88.3 (+4.5) | 88.8 (+4.2) | 80.9 (+3.2) |
Against the strongest prior video agents (VideoSeek/VideoARM 81.2 on VideoMME long, LensWalk 78.3 on VideoMMMU, VideoARM 76.4 on LongVideoBench long), the Gemini 3.1 Pro variant is +7.1, +10.4, and +4.5. Swap the backbone with no training: Kimi K2.5 +3.5, MiMo-V2-Omni +3.7, four-model average +4.2 on VideoMME (long).
Memory ablations on Gemini 3 Flash, VideoMME (long), one run each: native 80.7, append-only agent 81.9 (+1.2), dual-loop without the filesystem 83.3 (+1.4 more), full VideoLoop 85.8 (+2.4 more). On the hardest quartile Q4 the full system is 80.0 vs 70.7 append-only, a 9.3-point gap, 7.6 of it from the filesystem.
A blind judge (Gemini 3.1 Flash-Lite) reads a frozen context snapshot with no video and no tools, as a proxy for whether evidence is still recoverable. On easiest Q1, append-only retrievability is 86.7% vs 94.7% for VideoLoop. On hardest Q4 it falls to 60.9% vs 81.1%. Append-only curves plateau early and drop on Q1.
Tokens: full VideoLoop uses 618.2K per question, append-only 614.9K, +0.5% for +3.9 points. Dual-loop without the filesystem is more expensive at 647.0K (+5.2%). The filesystem cuts input from 584.6K to 559.0K. VideoLoop inspects fewer frames, takes fewer iterations, and spreads looks across the timeline; append-only piles up at the start and grows a long visual tail.
This applies a simple rule to long-video agents: curate memory, do not accumulate it. No training. The backbone is plug-and-play. What transfers is a forced rewrite of a bounded document after every step, with raw material on a filesystem.
The accuracy lift is incremental, 3 to 5 points on already strong Gemini 3.x models. The split that matters is hard questions and memory quality. Append-only multi-step search is only +1.2 over single-pass inference, so extra steps with a muddy memory barely pay. Extra tokens for the full system are negligible.
Use it on 30-60 minute videos where evidence is scattered and multi-hop. Skip it when the clip already fits in context.
There is no Limitations section. The conclusion claims a general principle for long-horizon agents and does not list failure modes. The thrashing argument is framed as a conceptual diagnostic, not a theorem reduced from OS paging. The rewrite inequality is a quality condition.
Ablations are one run per setting, no variance. Difficulty quartiles were sorted by Claude Opus 4.8 then checked by humans. The blind judge is also Gemini-family, so it may favor that lab's context style. A 32K budget and a 6-frame cap leave little visual bandwidth in working memory; even VideoLoop's Q4 judge still misses about 19%. Each step adds an extra LLM call. The paper reports token totals, not wall-clock or dollars, and does not cost-compare against VideoSeek or VideoARM.
Evaluation is almost entirely multiple choice, two of three benchmarks with official subtitles. Open-ended, subtitle-free, and multi-hour settings are absent. Part of the +7.1 over prior agents is the Gemini 3.1 Pro backbone already at 83.8, above VideoARM's 81.2. Same-backbone agent-versus-agent comparison is missing.