Video-RSI: Video Understanding Agents Recursively Self-Improve Their Harness
Bingjun Luo · hf · 2026-10-12
Video understanding agents acquire evidence through an executable harness that controls what they observe and how they use it, but execution traces only contain evidence from the current harness, leaving competing failure explanations unresolved. The authors introduce Video-RSI, a framework for recursive self-improvement in which the agent uses its own language model to revise its harness.
- Active video investigation: the model revisits original training videos to test competing failure explanations with additional observations, grounding proposed changes beyond the existing trace.
- Cost-aware harness evolution: diagnoses become reusable revisions, and which revisions to keep is decided by both answer accuracy and visual cost.
On video understanding benchmarks, the evolved agent improves accuracy while processing fewer frames, achieving competitive accuracy-efficiency trade-offs. Code is open-sourced.
More from coding & agent
- "I don't want to look at code, just state machines and live docs": one giant diagram — _AustinCalvert_ · 2026-10-12
- VectorDash: open-source Windows 11 search replacement powered by EmbeddingGemma 2 — andy2na · 2026-10-12
- Using AI agents to reverse engineer and patch games with byte-exact verification — arpit_bhayani · 2026-10-12
- Power user burns 86% of Codex weekly quota in 25 hours, complains resets can't keep up — AlchainHust · 2026-10-12
- dsco: an agentic LLM CLI written in C with tool use, swarms, and MCP — arthurcolle · 2026-10-12
- RLHF sawed off 'Decline': why LLMs hallucinate APIs and the one-line Flores fix — RandalSchwartz · 2026-10-12