VideoNexus: perception sub-agents return text evidence; 88.6% on LVBench at 44% of agentic cost

VideoNexus: Video Perception Beyond the Reasoning Context

Yancheng Long, Xuanyu Zheng, Wei Chen, Luoyuan Zhang, Haoyang Xu, Bin Wen, Tingting Gao

cs.CV

2026-11-17

Training-free agent whose planner never sees video: perception sub-agents return compact timestamped evidence. 88.6% on full LVBench, +1.1 over Gemini's agentic mode at 44% of its cost.

What problem this solves

An hour of video at one frame per second is well over a hundred thousand visual tokens; the 61-minute sample the paper measures comes to 159,576 for Gemini. Today's long-video agents leak context along two routes. Agents that pull frames or clips into the planner stack pixels into the same context that guides every later step, and the paper measures this coupled design at 1.47× the planner input of a decoupled one. Agents that delegate perception and take back text (VideoSeek, VideoXAgent) fix the pixels but not the scaling: per-frame captions only swap the constant, and in the authors' re-run VideoSeek's global scan returns one description per frame, 28 tokens for every frame read, so doubling the scan budget doubles the report. The question narrows to this: how do you let an agent look closely without every observation growing with sampling density?

Method

VideoNexus comes from KlingAI Research (Kuaishou) with CUHK and HKUST, and it needs no training. The core is a Planner that never touches the video stream, plus a set of short-lived perception sub-agents.

The design decision that matters: report size is decoupled from sampling density. In a coupled agent the observation scales with duration times frame rate, in pixels or in captions; here the planner retains only the compact report, whatever frame rate the sub-agent read. On the paper's worked example, perception burns 162.8k tokens while the planner sees 982 tokens of findings plus one card. The whole harness runs on a stock agent runtime with Model Context Protocol tools; Planner and sub-agents share one model, no weights touched.

Results

Full LVBench (1,549 multiple-choice questions, 103 hour-scale videos), all arms on the same Gemini 3.8 Flash; “Static” means feeding the whole video in one call:

MethodAccuracyCost/qFrames/q
Blind (question only)40.5%$0.0140
Static ingestion85.0%$0.1722,035
Gemini Agentic87.5%$0.319464
VideoSeek (re-run)81.4%$0.188123
VideoNexus88.6%$0.140621

+3.6 over Static (p=5×10⁻⁷) at lower cost; +1.1 over Gemini Agentic at 44% of its cost, p=0.10, not significant; +7.2 over the VideoSeek re-run (p=5×10⁻¹⁶), though that arm has no audio path, so it is a system-level comparison.

The mechanism measurements carry more weight than the leaderboard rows. Overview reads 1,019 frames plus the full audio track (about 141k media tokens) and returns 740 tokens, a 191× reduction, with a log-log slope of 0.00 against perceived volume; Inspect slopes at 0.18, its frames read rising 16.8× across deciles while returned text rises 1.67×. A paired ablation on 200 questions swaps returned pixels for Findings: planner input drops 32% (18.1M to 12.3M tokens) and accuracy moves from 86.5% to 89.0% (p=0.23, not significant). Evidence cards add +1.0 on Gemini 3.8 and +5.0 on Gemini 3.0, while stacking global-scan frames on top costs Gemini 3.0 four points. Squeezing sampling to VideoSeek's budget (117 versus 621 frames per question) drops accuracy from 90.0% to 87.0% at slightly higher cost: the perception saved comes back as extra rounds of interaction.

Coverage explains why density pays. The median annotated evidence window is 17 s and 30% are at most 5 s; a one-frame-per-minute uniform scan reaches only about half the questions, while at Overview's budget two or more frames land inside 71% of windows. Across planner models the margin shrinks as the baseline strengthens: +6.4 on Gemini 3.0 Flash, +6.5 with Kimi K3 on a 200-question subset, +3.6 against the 85.0% static run. Of VideoNexus's 177 errors, 122 are also missed by both Static and Gemini Agentic.

Why it matters

For practitioners the ordering is cost and context control first, accuracy second. $0.140 against $0.319 per question on hour-long videos, and nothing in the shared-backbone comparison is both cheaper and more accurate. The cost gap decomposes cleanly: VideoSeek emits 26.2k output tokens per question against 12.4k here, output bills at five times input, and that line alone settles the bill; model calls run 9.7 per question against 18.6. The transferable lesson sits at the interface: what a tool returns to the reasoning context matters as much as what it selects. Compact evidence exchange needs no training, accepts any model in either role, and ports to any agent runtime. On accuracy, keep expectations flat: the 1.1-point lead over Gemini Agentic is not significant. This is same accuracy at roughly half cost, not a capability jump.

Limitations

The authors disclose most of it. The Gemini Agentic gap is not significant (p=0.10), and the closed baseline prevents isolating where its cost comes from. The VideoSeek comparison mixes audio access with media pricing; repricing its images at the cheapest tier pulls its cost to exactly $0.140. The 8.5-point Video-GAIA gain sits on the 271 questions the team developed its web and crop tools on, judged by a single judge off the official three-run protocol, in a measured run where the crop tool read bounding boxes in the wrong frame; the authors call it a transfer result rather than a calibrated margin. A reader would add: most ablations run on 200 questions with p-values of 0.11 and 0.23 in a row; the +1.0 card gain on Gemini 3.8 has aggregate counts only and no paired test; there is no comparison against trained observation policies (EVA, VideoTemp-o3); and everything is a single execution, while the paper itself cites REPAIR on how much agentic video QA varies across re-runs. Read 88.6% as one well-controlled measurement, not a reproducible leaderboard entry.

Terms

Source

Related papers

All paper explainers